Saturday, December 18, 2010

Compress and split large files under linux

One of the common problems we used to face is transferring large files from Linux through mediums that doesn't support large files, especially when it comes to more than 4 GB files, usually that is a virtual machine hard disk, or backup file.

To compress and split files under linux, this command can be used:

$ tar -cvj big-file.dvi | split -b 650M -d - ""

This command will compress and split the "big-file.dvi" to many files with size 650MB for each, giving it the name ",, etc...."

To join the files together again, this command can be used:

$ cat* >

and to extract the output file :

$ tar -xvj

Sunday, November 14, 2010

rsync - The Linux Backup Command

rsync stands for remote sync.

rsync is used to perform the backup operation in UNIX / Linux.

rsync utility is used to synchronize the files and directories from one location to another in an effective way. Backup location could be on local server or on remote server.

Important features of rsync

  • Speed: First time, rsync replicates the whole content between the source and destination directories. Next time, rsync transfers only the changed blocks or bytes to the destination location, which makes the transfer really fast.
  • Security: rsync allows encryption of data using ssh protocol during transfer.
  • Less Bandwidth: rsync uses compression and decompression of data block by block at the sending and receiving end respectively. So the bandwidth used by rsync will be always less compared to other file transfer protocols.
  • Privileges: No special privileges are required to install and execute rsync


$ rsync options source destination

Source and destination could be either local or remote. In case of remote, specify the login name, remote server name and location.

Example . Synchronize Two Directories in a Local Server

To sync two directories in a local computer, use the following rsync -zvr command.

$ rsync -zvr /opt/lampp/htdocs/site1/ /backup1/site1bkp
building file list ... done
sent 26385 bytes received 1098 bytes 54966.00 bytes/sec
total size is 44867 speedup is 1.63

In the above rsync example:

  • -z is to enable compression
  • -v verbose
  • -r indicates recursive

Tuesday, October 12, 2010

What is Crontab ?

What is Crontab?
Crontab (CRON TABle) is a program that manipulates the CRON daemon, making it easy for users to schedule task and run programs/commands at pre determined periods of time. Crontab can also be considered a file witch contains commands that will be run by the system as the user that owns the crontab file.

What is the purpose of Crontab?
Cron is designed to maintain a list of commands that the system needs to run at a given time interval.

For example if you have a script that generates statistics and needs to be run every couple of hours or everyday cron can do it for you. Or for example if you have a script that sends a newsletter every month you can use cron to run the script that sends the newsletter every month or week.

Crontab commands
When your logged in to your server you can use program cron using the following commands:

crontab -l
Lists the current cron jobs
crontab -e
Edit your current crontab file and ad/remove/edit crontab tasks.
crontab -r
Remove the crontab file.
crontab -v
Displays the last time you edited your crontab file.

The crontab file - components of crontab

When you enter the edit mode (crontab -e) and start adding tasks to your cron file you need to consider the following syntax:

The asterisk (*) symbolizes that every instance of that field (i.e. every minute, hour, day, month, weekday) will be used in the command.

Example on how to setup your first crontab
Lets say you have a script named located in /home/your_username you want to run every day at 13:30. You will need to login your server console and input the following commands:

crontab -e

This will open the crontab file and let you edit it. By default this file will be opened with the VI editor and you will need to press the "Insert" key on your keyboard to be able to write in that file.

30 13 * * * /home/your_username/ >/dev/null 2>&1

The first character you see is "30" this means that crontab will run the script every time the clock hits the 30 minutes mark. Next "13" this means that crontab will run the script when the clock hits 13. The next three * tell crontab to run the script every day, of every month of every weekday. Combining these fields crontab will run the script every day at exactly 13:30. You may notice that we added the ">/dev/null 2>&1" string at the end of the command. The default cron job will always send and e-mail to the root account when ever a command is executed. Now you don't want to be notified every day that your crontab job has been executed. If you don't want to receive e-mails every day notifying you about your job's execution place this ">/dev/null 2>&1" at the end of every instance of every crontab command.

Now to list your crontab job just issue the following command: crontab -l
If you need to ad another crontab job or even more all you need to do is follow the same steps as above and just ad another line to the file.

REMEMBER: Every crontab job needs to be placed on its own line in the file and after the last line you need to insert a non-braking character (press Enter).

The crontab syntax goes a little beyond its boundaries and has more advance meaning for some users.
For example if you wish to use more then one instance of one column you will separate those instances by a comma "," or if you wish to use a time period lets say for example from Sunday till Tuesday you will use dash "-".

10,20,30 13 * * 0-2 /home/your_username/ >/dev/null 2>&1

This crontab job will run your scrip "" on from Sunday until Tuesday (Sunday, Monday, Tuesday) at 13:10, 13:20 and 13:30. Remember there are no spaces between values separated by commas "," and neither in dashes "-". There is also an operator that some versions of cron support called the slash operator "/" that can be used to skip a given number of values in your jobs.

Crontab examples - cron examples
The syntax of crontab is not very easy to understand from the start and the best way of understanding is from examples:

Run the script every day at 12:00.
0 12 * * * /home/your_username/ >/dev/null 2>&1

Run the script every day at 12:00, 14:00 and 16:00.
0 12,14,16 * * * /home/your_username/ >/dev/null 2>&1

Run the script every Sunday at 13:30.
30 13 * * 0 /home/your_username/ >/dev/null 2>&1

Run the script every Saturday at 12:00, 14:00 and 16:00.
0 12,14,16 * * 6 /home/your_username/ >/dev/null 2>&1

Run the script on the first, fifteenth and twentieth of every month.
0 0 1,15,20 * * /home/your_username/ >/dev/null 2>&1

Run the script every day from 3 to 3 hours: 00:00, 03:00, 06:00 etc.
0 */3 * * * /home/your_username/ >/dev/null 2>&1

Thursday, July 29, 2010

Install JSON PHP Extension on RedHat / Fedora

To enable JSON Support in php, do the following in Redhat / Fedora Server

  1. Execute the following Command from root account
    # pecl install json
  2. Ensure that exists in folder /usr/lib/php/modules
  3. If exists, add the following line to /etc/php.ini
  4. Restart Apache (# service httpd restart)
  5. Execute phpinfo() to verify JSON enabled

Saturday, June 12, 2010

Virtual Hosts on Apache

Virtual Hosts on Apache
The term Virtual Host (Vhost) refers to the practice of hosting a number of websites on a single webserver, serving diffrent content for each website.

Lets say for example you needed to setup a webserver for your companys website. Now asume that your company decides it needs to host a second website. In this scenario one may asume that you need to go buy and set-up a new web-server. That is not the case, with Apache's Vhost functionality it becomes a easy task to setup more the one website on the same server.

Understanding how Apache works
Before one can actually start seting up Vhosts its a good practice to understand the way they work and how Apache handles the process.

When a user is typing an URL address in the web browser and hits the Go button that address is translated to an IP address used for locating the webserver. That is a request and its being sent to the webserver for processing. When the Apache webserver recieves this request it checks the headers to determin the name of that address. This is the most important thing in the whole process, Apache needs to know the actual address.

When Apache is started on Linux, it loads its configuration file (usually located in /etc/httpd/conf/httpd.conf). This is the main configuration file for Apache, and this is also the place to add Virtual Hosts. Reading the httpd.conf file, Apache creates a list of its IP addresses and Virtual Hosts.

When Apache recieves the request for a web page it checks its lists to determine if a Virtual Host for that domain exists and what directory it should serve. If there are no VHosts for that domain, it will serve the default DocumentRoot of the server, usually locate in /var/www/html. However, if Apache is configured with VHosts and a ServerName match is found, it will server the containing files of the folder specified in the VHost.
Creating a basic Virtual Host
After you get your Apache server up and running you can start setting up the VHosts. Using your favorite text-editor openup the httpd.conf file usually locate in /etc/httpd/conf/httpd.conf.
Lets say you are using the VI editor, type the following command:

vi /etc/httpd/conf/httpd.conf

You will notice a large file, filled with configuration variables. Scroll down to the end of the file (you can do that faster by pressing Page_Down) untill you find the following text:

# VirtualHost example:
# Almost any Apache directive may go into a VirtualHost container.
# The first VirtualHost section is used for requests without a known
# server name.
# ServerAdmin
# DocumentRoot /www/docs/
# ServerName
# ErrorLog logs/
# CustomLog logs/ common

Ther you have it, an official VHost example, located in the httpd.conf file.
The "#" charactes in front of every line are comment-out characters this means those lines dont affect the Configuration, they are there just for helping purposes.

As you may notice a new term called DIRECTIVE is mentioned. A directive are Apache commands (or variables) processed by the server. The directives in this case are: ServerAdmin, DocumentRoot, ServerName, ErrorLog and CustomLog. This directives are the main ones we are going to use and they are self explanatory.

Now lets asume that your main website is named and the second website is Your main VHost configuration will be like this:

Listen 80

DocumentRoot /var/www/html
ErrorLog /var/log/httpd/example-com.log

DocumentRoot /var/www/
ErrorLog /var/log/httpd/website2-com.log

Dont get scared, we will clear everything up right now.Listen 80 - this will tell Apache to listen only on port 80.

NameVirtualHost - this is the name of the main Virtual Host, your main web-server name (this could also be replaced by the server's IP address or by the * charachter).
- this is the starting tag of the first VHost, your main server domain, that will be served when a user types If for example Apache recieves a request for a domain that is not matched in the VHosts, it will serve files from the first VHost, This field could also be replaced by , wich basicly means the same thing but I prefer keeping things organized in my http.conf file and I always use this notation.
ServerName - this is the main directive of Apache VHosts. When Apache recieves a request for a domain, that domain name will be searched in this lists of VHosts.
DocumentRoot /var/www/html - if the ServerName is located in the VHost, Apache will start serving files from this directory, thus if is requested, files from /var/www/html will be served.
ErrorLog /var/log/httpd/example-com.log - this is the place where error logs will be stored. Its a good practice to keep your error logs separated for everydomain, thus when you will encounter a problem with a website you will only need to debug a single error log file.
< /virtualhost> - the ending tag of a Virtual Host.
In order for the VirtualHost to take effect the server (httpd) needs to be restarted or reloaded.
Use the follwing command to restart Apache:
Service httpd restart