Thursday, April 30, 2009

Disable Users to login into the server

Suppose we want to take a backup of user's account. So for that first of all we have to block the user to login into their account to maintain data integrity of user's backup files. So using below technique we can do that very easily.enjoy !!!


Edit the pam file located in /etc/pam.d/ directory for the service you want to control.


 Example : Suppose you want to do control  ssh service


Step 1: Add below line in /etc/pam.d/sshd file if it is not available.


account required pam_nologin.so


Step 2: Create the /etc/nologin file,


# touch /etc/nologin


This should disable the login from ssh for every user except administrator user(root).


Step 3: To re-enable the login just remove /etc/nologin


# rm –rf /etc/nologin

Split the Large files

Split on a 300mb example.zip file:


#split -b 100mb example.zip


It will generate 3 files with the following file sizes:



100MB xaa
100MB xab
100MB xac

After split use: cat to combine a file


#cat xa* > example-new.zip 


 

Open port using iptables

To open a required port you have to know the basic information. 

1) Service name ( ssh,ftp,etc...)

2) port number ( 22,25, etc...)

3) tcp port or udp port (tcp or udp)

Example : To enable ssh access to your Server from anywhere

#iptables -A allowed -p tcp -dport 22 -s 0/0 -j ACCEPT

#iptables -A allowed -p udp -dport 22 -s 0/0 -j ACCEPT

Time bases iptables rules

If you want to restrict/allow access to certain service on timely basis using iptables. 


Use : iptables patch-o-matic extension (pom or p-o-m)


That allows us to match a packet based on its arrival or departure  timestamp.


Syntax : iptables RULE -m time –timestart TIME –timestop TIME –days DAYS -j ACTION


                –timestart TIME: Time start value (format is 00:00-23:59)


               –timestop TIME: Time stop value (the same format)


               –days DAYS: a list of days to apply, from (format: Mon, Tue, Wed, Thu, Fri, Sat, Sun).


Example :  We want to  restrict access to SSH between 10:00 pm - 8:00am on weekdays.


#iptables -A INPUT -p tcp -d 192.168.10.1 –dport 22 -m time –timestart 22:00 –timestop 8:00 -days Mon,Tue,Wed,Thu,Fri -j DROP


Enjoy !!!!!!!!!!!!

Disable a User Account in Linux

Technique 1 : Using /etc/shadow file

Linux systems use /etc/shadow to store the encrypted user passwords.

Active user account will have one line in /etc/shadow 

username:$2$eF7dafdsf$4kfdsm$3Fkm6nl.:13852:0:99999:7:::

Here second field is the encrypted password.


If we replace the password with “*” or “!” this will make the account unusable, and it means that no login is permitted for that user.


username:*:13852:0:99999:7:::


But disadvantage of this technique is password will be loss and we have to generate a new password for this user.


Technique 2 : Using passwd -l


It will lock the user account. After type passwd -l it will give you "password changed" message. This command will do the changes in the /etc/shadow file and add the "!" in the second field of that user.



username:!$2$eF7dafdsf$4kfdsm$3Fkm6nl.:13852:0:99999:7:::

if we want to unlock the user account then we can do that after removing "!" from /etc/shadow file. We can also do that mannual by using passwd -u command.




change the login banner/message

The login banner is essential to legal actions taken against people who misuse and illegally hack into your box. 


 To change the login banner/welcome message, Edit the /etc/issue file and put whatever you want into this file and save the file and exit.


 #vi /etc/issue

Duplicating a disk

If you have two IDE drives that are of identical size, and provided that you are sure they contain no bad sectors and provided neither are mounted, you can run


dd if=/dev/hda of=/dev/hdb


To copy the entire disk and avoid having to install an operating system from scratch. It doesn’t matter what is on the original (Windows, LINUX, or whatever) since each sector is identically duplicated; the new system will work perfectly.

How to Prevent the reuse of old passwords

For RHEL/Fedora distribution 


To remember the last 5 passwords, add the line below to the file /etc/pam.d/system-auth file:


password sufficient /lib/security/pam_unix.so use_authtok md5 shadow remember=5


For Debian/ubuntu distribution


To remember the last 5 passwords, add the line below to the file /etc/pam.d/common-password file:


password sufficient /lib/security/pam_unix.so use_authtok md5 shadow remember=5

BackUp and Restore MBR after Windows Crash

Step 1 : Login as a root user.


# dd if=/dev/hda of=/dev/fd0 bs=512 count=1




This makes an exact copy of the MBR of the first hard drive (hda - you need to replace this), copying it to a floppy disk. You can boot directly from this floppy, and see your old boot menu. You can restore it by switching the “if=” and “of=” (input file, output file) parameters.


 




If you don’t have a floppy drive, you can back it up to a file using below command.


# dd if=/dev/hda of=/home/Username/boot.mbr bs=512 count=1


Use your Linux distribution’s installation CD to boot into rescue mode and restore it with below command.


 # dd if=/mnt/hda5/Username/boot.mbr of=/dev/hda bs=512 count=1



How to Use MD5 Sum

Full Name : Message-Digest algorithm 5

Usage : Using an MD5 checksum you can  verify the integrity of data

Algorithm : cryptographic hash function with a 128-bit value 

 


MD5 sum first identify the the data which is backup and then create a MD5 checksum which is combination of unique string of letters and numbers put together string like : 3dfsdjl2342ldkfjkdf32k. MD5 checksums are very useful for the verification of data and for passwords



 

Check MD5 sum usage

Step 1 : Generate a MD5 checksum:

 Go to in Shell console

#md5sum filename > filename.md5″


Step 2: Verify a MD5 checksum:

#md5sum -c filename.md5″

check SSL certificate expiration

Usage: ./check_expiration www.verisign.com


[-] Certificate for www.verisign.com has not expired yet: May 8 23:59:59 2009 GMT


 


#!/bin/bash 


dates=$(echo "GET /" |openssl s_client -connect "$1:443" 2> /dev/null |openssl x509 -noout -dates) 


# For STARTTLS over e.g. smtp, replace the offending field by: 


# openssl s_client -connect "$1:25" -starttls smtp if [ -z "$dates" ]; then    


 echo "[!] Invalid IP, not SSL or no cert found"     


exit 2 


fi 


not_after=$(echo $dates|cut -d '=' -f 3) 


now_epoch=$(date +%s) 


not_after_epoch=$(date +%s -d "$not_after") 


if [ $now_epoch -gt $not_after_epoch ]; then    


echo "[!] Certificate for $1 has expired: $not_after"     


exit 1 


else     


echo "[-] Certificate 


for $1 has not expired yet: $not_after" fi





After Completing the script change the permission of the script


#chm0d +x check_expiration.bash

Iptables Unblock/delete

Display the List existing chains Entry


iptables -L –n


iptables -L -n -v


iptables -L chain-name -n -v


iptables -L spamips -n –v


 


Display List existing chains with line number







iptables -L INPUT -n --line-numbers


iptables -L OUTPUT -n --line-numbers


 iptables -L spamips -n -v --line-numbers


 


Delete Rule from IPTABLES using line number


iptables -D INPUT linunumber


Example : iptables -D INPUT 11


 


You can also use the Below syntax to delete / unblock an IP 


iptables -D INPUT -s ipaddress -j DROP


 



 

Setup Automatic Users Logout After a Period of Inactivity

* For SH/BASH/TCSH


Below script will implement a 5 minute idle time-out for the default /bin/bash shell.



Step 1: Create file autologut.sh

 # vi /etc/profile.d/autologout.sh


Append the following code:


TMOUT=300


readonly TMOUT


export TMOUT


Step 2: Save and close the file. Set permissions:


# chmod +x /etc/profile.d/autologout.sh



 

*For tcsh version follow the below process


Step 1: Create file autologut.sh



# vi /etc/profile.d/autologout.csh
Append the following code:

set -r autologout 5


Step 2: Save and close the file. Set permissions:


# chmod +x /etc/profile.d/autologout.csh


 


For SSH user inactive interval we can define that in configuration file of the ssh server


Step 1 : Configure /etc/ssh/sshd_config file


# vi /etc/ssh/sshd config


Find ClientAliveInterval and set to 300 (5 minutes) as follows:


ClientAliveInterval 300


ClientAliveCountMax 0



Step 2: Save and close the file. Restart sshd:
# service sshd restart

 

Setup pkgutil in Solaris

 Step 1 - Install pkgutil


 


 Solaris 8 & 9 Users : You need to fetch the correct pkgutil package from Network.com thus : 




      md5 = 7263f7010b15899bcf9bb7014c43ff7b
      sha1 = 0e9b56018796718b824de0c4a1cdf4fccb4a4087
      sha256 = d5d3746c0e981b69a102862352b1db7510281a94c2b82c1003a2f0a15f3b1e61


      md5 = 029b42a0dc9653959bf29b6f68dec8d4
      sha1 = b67bf038ca4b33a18e11e7bf7dee48f62b4b6ce9
      sha256 = 862513f65d143fd87e8d0dc2db611a2f7a587d94d00f2e3b25b04d958666ab27

#pkgadd -d pkgutil_i386.pkg


# mkdir /etc/opt/csw


# cp -p /opt/csw/etc/pkgutil.conf.CSW /etc/opt/csw/pkgutil.conf


 


Solaris 10 & Nevada or OpenSolaris™ Users : 


#pkgadd -d http://blastwave.network.com/csw/pkgutil_`/sbin/uname -p`.pkg


# mkdir /etc/opt/csw


# cp -p /opt/csw/etc/pkgutil.conf.CSW /etc/opt/csw/pkgutil.conf


 


Step 2 - Fetch the Software catalog


#/opt/csw/bin/pkgutil –catalog 


 


 Step 3 - Security First !


#/opt/csw/bin/pkgutil –install gnupg textutils 


## cd /tmp


#/opt/csw/libexec/pkgutil/wgethttp://www.blastwave.org/gpg_key.txt


# /opt/csw/bin/gpg –import gpg_key.txt


1) Verify that you have the key. Use the gpg –list-keys command :


# /opt/csw/bin/gpg –list-keys.


2 ) Mark the key as being trusted for a given purpose. In this case you want to trust the signed software catalogs from Blastwave. To do this you need to edit the key and mark it as being trusted thus : 


# /opt/csw/bin/gpg –edit-key A1999E90


Command> Trust


Your decision? 5


Do you really want to set this key to ultimate trust? (y/N) y


Command> quit


3) Lastly you need to modify the pkgutil configuration to actually use this GPG key as well as to verify the MD5 hashes of the software packages. Edit the pkgutil.conf file that you copied into /etc/opt/csw such that the lines related to GPG and MD5 are not commented out. Your pkgutil.conf should look like so : 


use_gpg=true


use_md5=true


# /opt/csw/bin/pkgutil –catalog 


#export PATH=/opt/csw/bin:/usr/sbin:/usr/bin:/usr/dt/bin:/usr/openwin/bin:/usr/ccs/bin


 


#pkgutil -a                                         list available package


#pkgutil -i     packagename    install package


 


If you find any difficutly then refer below link


http://www.blastwave.org/howto.html

Backup script to copy directories from Linux server to Windows

There are two files contains in this script.

1) Backup Script (backup.pl)

2) Config file ( file.config)

Backup.pl contains the script to copy data from Linux Server to Windows server. Config file contains the information regarding to which directory to copied from linux server to windows server. It contains the list of directory which we are going to take backup.

 

file.config

/etc/apache2            etc_apache/

/etc/hosts etc_hosts/

/etc/sysctl.conf etc_sysctl/

/etc/resolv.conf etc_resolv/

/etc/php5/apache2/php.ini etc_php5_apache2_php/

/etc/dhcp3 etc_dhpc3/

/etc/network etc_network/

/etc/bind etc_bind/

/var/www/trac var_www_trac/

/root/backupscript root_backupscript/



backup.pl 

-----------------------------------------------------------------------------------------------

#!/usr/bin/perl

use File::Basename;


`mount -a`;

$backupfolder = '/home/Backup';     # 

$config_file = '/path/to/file.config';   #config file location

my @directory = getFileContents($config_file);

chdir($backupfolder) or die ("Cannot go to folder '$backupfolder'");




foreach my $folder_locations (@directory) {

                my($folder,$backup_location)= split(/\s+/,$folder_locations);

                `rsync -azvL $folder $backupfolder/$backup_location`;

 }




sub getFileContents {

        my $file = shift;

        my @lines;

        if (!open (FILE,$file)) {

                        die("Can't open '$file': $!");

        } else {

                @lines = <FILE>;

                close(FILE);

        }

        return @lines;

 }
-----------------------------------------------------------------------------------------------

Wednesday, April 29, 2009

MySQL Reset Root Password

We can reset root Password using two techniques. First one is using the mysqladmin command and second one is using the mysql safemode technique. I have describe both the technique below. enjoy !!!!!!!!

Technique 1. Using Mysqladmin command

#mysqladmin -u root -p password 'new-password'

Technique 2. Using Mysql safe mode 






# /etc/init.d/mysql stop


# mysqld_safe --skip-grant-tables  &


# mysql -u root


mysql> use mysql;


mysql> update user set  password=PASSWORD("newpassword") where User='root';


mysql> flush  privileges;


mysql> quit


# /etc/init.d/mysql stop


# /etc/init.d/mysql  start



Log file rotation using /etc/logrotate.conf

Step 1: /etc/logrotate.d directory

#cd /etc/logrotate.d

Step 2 : Create a file apache

#touch apache

/var/log/apache2/*.log {

        weekly

        missingok

        rotate 52

        compress

        delaycompress

        notifempty

        create 640 root adm

        sharedscripts

        postrotate

                if [ -f "`. /etc/apache2/envvars ; echo ${APACHE_PID_FILE:-                /var/run/apache2.pid}`" ]; then

                        /etc/init.d/apache2 reload > /dev/null

                fi

        endscript

}
Description of  parameters:




  • weekly : Log files are rotated if the current weekday is less then the weekday of the last rotation or if more then a week has passed since the last rotation.

  • rotate 52 : Log files are rotated 52 times before being removed

  • compress : Old versions of log files are compressed with gzip to save disk space.

  • missingok : If the log file is missing, go on to the next one without issuing an error message.

  • notifempty : Do not rotate the log if it is empty

  • sharedscripts : Single script may be run multiple times for log file entries which match multiple files. If sharedscript is specified, the scripts are only run once, no matter how many logs match the wildcarded pattern. However, if none of the logs in the pattern require rotating, the scripts will not be run at all.




Find Out Which Table is Consuming Resources using Mytop

Step 1 : Install required perl modules

cpan -i Term::ReadKey
cpan -i Term::ANSIColor
cpan -i Time::HiRes


Step 2 :  Install mytop in your system

#wget http://jeremy.zawodny.com/mysql/mytop/mytop-1.6.tar.gz

# tar -zxvf mytop-1.6.tar.gz

#cd mytop-1.6

#perl Makefile.PL

#make

#make test

#make install


Step 3 : To view information of database example run below command.

# mytop -u username -p 'password' -h Hostname -d example

Allow a normal user to run commands as root in Unix/linux

Use sudo command which is use to execute a command as another user.

The sudo command allows users to do tasks on a Linux system as another user.

Sudo is more  secure then su command.


Log files : /var/log/secure (Red Hat/Fedora / CentOS Linux)  


   /var/log/auth.log (Ubuntu / Debian Linux).


So we can get the details using above log files that which user has perform which task or command using sudo or as privilage user.



Below is the general syntax for /etc/sudoers file

 



USER HOSTNAME=COMMAND

Use of Sudo : 


1) #visudo


2 )Enter this line in open file  sanjay localhost=/sbin/reboot


3 ) Save the file


4 ) Now user sanjay can reboot the server using below command


# sudo /sbin/reboot


You can see the information of this user in /var/log/auth.log  or /var/log/secure file


 


Examples : 


1) Specify multiple commands for user Sanjay:


Sanjay ALL=/sbin/reboot, /etc/init.d/apache2 , /etc/init.d/mysql



2) Allow user Sanjay to run /sbin/reboot without any password 
Sanjay ALL= NOPASSWD: /sbin/reboot


3) Allow user sanjay to run any command from /usr/bin directory on the localhost:
sanjay localhost = /usr/bin/*

Open vim editor to first occurrence of search term

To open a file to the first instance of a search term, use the following command line with vim:


vi +/searchterm filename


You can search for the text “word1 word2” from the file filename.txt in at least two ways, vi +/word1 \ word2 filename.txt


vi +/"word1 word2" file.txt

Generate a Hardware Profile for Your Ubuntu System

1. Open a terminal window.


2. Execute the following command:sudo lshw -html > hardwareprofile.html


3. Close the terminal window when the command has finished.


4.  Open hardwareprofile.html file




Shell script to backup MySql database

#!/bin/bash


# Shell script to backup MySql database


 


USERNAME="mysql user name"    


PASSWORD="PASSWORD"


HOSTNAME="localhost" 


 


MYSQL="$(which mysql)"


MYSQLDUMP="$(which mysqldump)"


CHOWN="$(which chown)"


CHMOD="$(which chmod)"


GZIP="$(which gzip)"


 


DESTINATION="/directory1"    # Directoy to take backup


 


# Main directory where backup will be stored


BACKUPSTORE="$DEST/mysql"


HOST="$(hostname)"                                  # Get hostname


NOW="$(date +"%d-%m-%Y")"                  # Get data in dd-mm-yyyy format


FILE=""                                                           # File to store current backup file


DATABASELIST=""                                     # Store list of databases


IGNOREDATABASE="temp"                   # DO NOT BACKUP THESE DATABASES


 


[ ! -d $BACKUPSTORE ] && mkdir -p $ BACKUPSTORE || :


 


$CHOWN 0.0 -R $DESTINATION


$CHMOD 0600 $DESTINATION


 


# Get all database list


DATABSELIST="$($MYSQL -u $USERNAME -h $HOSTNAME -p$PASSWORD -Bse 'show databases')"


 


for db in $DATABASELIST


do


    skipdb=-1


    if [ "$IGNOREDATABASE" != "" ];


    then


            for i in $IGNOREDATABASE


            do


                [ "$db" == "$i" ] && skipdb=1 || :


            done


    fi


 


    if [ "$skipdb" == "-1" ] ; then


            FILE="$BACKUPSTORE/$db.$HOST.$NOW.gz"


            $MYSQLDUMP -u $USERNAME -h $HOSTNAME -p$PASSWORD $db | $GZIP -9 > $FILE


    fi


done

Set Up Database Replication In MySQL

Configure the MySQL Master Server


Step 1 : edit /etc/mysql/my.cnf file. 


#skip-networking


#bind-address            = 127.0.0.1


(add below line in /etc/mysql/my.cnf file)


server-id               = 1


log_bin                 = /var/log/mysql/mysql-bin.log


binlog_do_db            = replicationdb


step 2 :  Restart Mysql server


#/etc/init.d/mysql restart


Step 3 : create a user with replication privileges:


#mysql -u root -p


mysql> GRANT REPLICATION SLAVE ON *.* TO ’replicationuser’@'%’ IDENTIFIED BY ‘<some_password>’; 


mysql>FLUSH PRIVILEGES;


 mysql>USE replicationdb;


mysql>FLUSH TABLES WITH READ LOCK;


mysql>SHOW MASTER STATUS;


Result of above command:


+—————+———-+————–+——————+


| File          | Position | Binlog_do_db | Binlog_ignore_db |


+—————+———-+————–+——————+


| mysql-bin.005 | 180      | replicationdb    |                  |


+—————+———-+————–+——————+


1 row in set (0.00 sec)


 Please remeber above information will need in slave server configuration 


  mysql>quit;


Step 4 : Dump replicationdb from the Master server 


#mysqldump -u root -p<password> replicationdb > replicationdb.sql 


Above command create  dump of replicationdb in the file replication.sql. Transfer this file to your slave server!


Step  5 : Unlock the tables


#mysql -u root -p


mysql>UNLOCK TABLES;


mysql>quit;


 


Configure The Slave Server


Step 1 : Create the database replicationdb



#mysql -u root -p
mysql>CREATE DATABASE replicationdb;
mysql>quit;

Step 2 : Extract the dump database in replicationdb database


#mysql -u root -p<password> replicationdb < /path/to/replicationdb.sql 


Step 3 : Add the below lines into /etc/mysql/my.cnf


server-id=2


master-host=192.168.10.175


master-user=replicationuser


master-password=secret


replicate-do-db=replicationdb


Step 3 : Restart MySQL:


# /etc/init.d/mysql restart



Step 4 :  Allow slave user to connet to Remote Master server

# mysql -u root -p

mysql>SLAVE STOP;


mysql>CHANGE MASTER TO MASTER_HOST=’192.168.10.175′, MASTER_USER=’replicationuser’, MASTER_PASSWORD=’<some_password>’, MASTER_LOG_FILE=’mysql-bin.005′, MASTER_LOG_POS=180;


mysql>START SLAVE;


mysql>quit;


Configuration is Complete now!  So whenever replicationdb is updated on the master server, all changes will be replicated to replicationdb on the slave server.


Enjoy !!!!!

Upgrade Linux kernel

Upgrade of the kernel in Debian or Ubuntu Linux

# apt-cache search kernel-image   ( find the kernel image)

#apt-get install kernel-image-x.x.x-xx

Upgrade of the kernel in Fedora Linux / CentOS / RHEL 5

#yum update kernel

You can also download the rpm and execute the below command

#rpm -ivh kernel*

Upgrade of the kernel in Red Hat enterprise Linux version <= 4.x

if your system is registered with RHN support then

#up2date -f kernel

For SMP kernel (for multiple CPU) use the below command:

#up2date -f kernel-smp

Block brute force attacks using denyhost

Issue:

Check your /var/log/auth.log file. In this file you can able to see that some unauthorized user is constantly trying to access your system using brute force attacks. He/she trying to get the access of your system by some scripts which is trying to login with different password. We can block this using simple iptables rule.

Step 1 : Install Denyhost tools in your system.

apt-get install denyhost

Step 2 : Configure /etc/denyhost.conf file based on your requirement

* configure this file to anylize the log file

SECURE_LOG = /var/log/auth.log(default)

DENY_THRESHOLD_INVALID = 3 (will  block each host after the number of failed login)

DENY_THRESHOLD_VALID = 5  (will  block each host after the number of failed login for valid user login attempts)



DENY_THRESHOLD_ROOT = 3 (for root login)

Step 3 : Restat denyhost

#/etc/init.d/denyhost restart

I switched from Squid to Sun Java System Web Proxy Server

I’ve been running Squid Web Proxy Cache for quite a while and also documented some basic setup in another article. But the last time we set up a server I decided to try Sun Java System Web Proxy Server. Since then, I switched the remaining Squid servers to Sun’s proxy and lived happily ever after. 

Why? Well, Squid was giving me no problem but sometimes setting it up and managing it was boring and error prone. Sun’s Web Proxy Server has got the (familiar) administrator’s web interface and I practically never touch a configuration file by hand. Creating a basic setup it’s really a question of clicking a couple of button and the proxy’s up and running.

Installation.
Installation is pretty straightforward. I downloaded the Sun Java Enterprise System and launched the installer. Once launched, I just checked the Sun Java System Web Proxy Server and the installer did it all. The installer also gives you the possibility of automatically creating a proxy server with the default configuration values and if you need a good starting point that’s a good hint.

Creating a server.
This was easy too. I had to create two different web proxies because we’re serving two subnets with different requirements. Once the installer finishes its work, you can connect to the administration console using the configuration values you provided during the installation:

  • administration port

  • admin password


Open your favorite browser and launch the console. Once you’re in, you’ll find yourserlf in the Server/Manage Server section:

Adding a server is pretty easy, it just asks you for (very) basic information:



Inspecting default configuration.
Once you’re done with creating your server(s), you can inspect the default configuration with the Manage Servers/Preferences/View server settings option:



Configuring system preferences.
Using the Manage Servers/Preferences/Configure system preferencestab you can modify basic preferences for your proxy:

In this page you can set:

  • server user: by default, it’s nobody, and it’s a value I usually don’t need to change.

  • processes: the number of the background processes used to serve incoming requests.

  • listen queue size: the maximum number of pending connections on a socket.

  • request throttle: the number of concurrent transactions that the proxy can handle.

  • enable DNS: this is useful mostly for logging and for managing access control. If you enable DNS, the proxy will resolve IP into host names.


There are other configurable options, many of which are useful if you plan to implement distributed caching, whic I’ll not cover in this post.

Adding listen sockets.
The next thing you’ll probably want to do is setting up listen sockets, which are the endpoints of the proxy to which your clients will connect. If during the installation a default server was created for you, you’ll probably want to edit the default port value for the listen socket:

Setting up cache properties.
The last thing you’ll probably do to set this basic web proxy server is configuring the cache. You can start in the Manage servers/Cachesection of the admin application. The first panel is Set cache specificswhere you can set the most common properties for you cache.



The first thing I usually do is changing the cache working directory. Remember that when you change the cache directory you must pay attention that the proxy user (in my case nobody) can write into that directory, otherwise the cache won’t work.

One chosen your favorite directory, you can set up the cache capacity either with the provided drop down list or via the Cache capacity configurator.

In this page you can also configure basic caching behavior for HTTP, FTP and Gopher protocols. As far as it concerns the HTTP protocol:

  • Always check if the document is up to date: this option does exactly what it says: every time a document is requested to the proxy, the proxy will check that the version it is caching is up to date. This may be useful in some circumstances but will rise the number of outgoing connection from the proxy server.

  • Check only if last check more than: if you choose this option, the proxy server will open a connection to check if the document is up to date only if the last time it did was more than what you specify. The default is two hours and depending on the situation I use to rise it up to one entire day.

  • Using: this option controls how the proxy server checks if the document is up to date. You can choose either using the last-modification factor, which is the set of headers that the web server sends along with the document, or the explicit expiration information, which are the internal headers used by the proxy server.

  • Never report accesses to remote server: this option tells the proxy server not to report a cache hit to remote servers.

  • Report cache hits to remote server: this option tells the proxy server to report to the remote server the number of times a document has been hit in the cache and accessed from there. This option rises the number of outgoing connection from the proxy server and may hit latencies and performance.


Cache partitions.
The cache partitions are the parts of disk reserved for caching purposes by the proxy server. You’ll need to edit the cache partitions properties in the case, for example, you rise the cache capacity and you need to reserve more space on disk by adding a new cache partition.

In the previous screenshot the cache partition is 1.6 GB, which is the cache capacity I set up for this server. Adding a cache partition is trivial, you’re only asked about the directory which will host the partition.

Set garbage collection.
As long as you use the proxy server, it will cache documents you request and the cache will keep growing up maintaining the allocated space in the range specified by the caching configuration. The garbage collection is the process that cleans up documents from the proxy cache and must be performed periodically. By default, this property is set as Automatic. I observed in my proxy server instances that if the cache hits are high and you are caching big documents, even if the garbage collection is automatic, it seems to never take place and the cache keeps growing up. For this reason I suggest you plan and schedule regular gargabe collection cycles. You may schedule them via the system cron or via the internal proxy scheduler. I usually use the system cron. Once chosen the manual configuration option, explicit garbage collection cycles can be scheduled in the Schedule garbage collection panel.

Caching configuration.
Other useful options you may want to set up can be found on the Set caching configuration panel. By default, the caching default is thederived configuration. If you want to explicitely set up every option, you can then set cache as the caching default value. Once done that and pushed the OK button, a new form will appear:
 

The options you’ll find usually are:


  • The cache default

  • How to cache pages that require authentication

  • How to cache queries

  • The minimum and maximum cache file sizes

  • When to refresh a cached document

  • The cache expiration policy

  • The caching behavior for client interruptions

  • The caching behavior for failed connections to origin servers


An option which is often overlooked and might be pretty important for your proxy performance are the last two which rule what happens when a proxy connection is broken. This may happen if, for example, your user exits the browser or cancel a connection: the proxy may continue downloading the entire file even if the client is not retrieving it any more and this effect may sum up when many client are connected leading to proxy saturation and lost of performance. I saw this happen many times, even if with multimedia content such as flash-based solutions which deliver content, like YouTube. For this reason, I usually set 100% for the caching behavior for client interruptions which in effect has the proxy close the remote connection whenever a client disconnects.

Conclusion.
With just few and simple steps you’ve set up an enterprise grade web proxy server. I suggest you to check the official documentation at Sun documentation center to fine tune your setup and read about more advanced configurations such as connecting to an LDAP to authenticate users, setting up SOCKS and setting up proxy arrays for distributed caching.

Now, enjoy your new proxy server!

Configuring NTP server and client on Solaris 10

One of the task which is often performed during the setup of a machine it’s the setup of the NTP daemon.

NTP is one of the oldest internet protocol still in use and it allows the synchronization of computer clocks distributing UTC (Coordinated Universal Time) over the network. NTP design is focused on compensating the variable latency of the network.

A machine may be an NTP client or an NTP server. Roughly, an NTP client it’s a machine that uses the NTP protocol to synchronize its clock and an NTP server it’s a machine that provides NTP client the information needed to set their clocks and itself uses other NTP server to keep in sync.

NTP on Solaris 10
Solaris 10 ships with an NTP daemon, ntpd, configured via SMF (svc:/network/ntp:default) and a bunch of sample ntp.conffiles to quickly configure a machine as a client or as a server.

# svcs ntp
# svcs ntp
STATE STIME FMRI online 0:43:33 svc:/network/ntp:default

Configuring a client
If your machine is just a client, you can just pick the/etc/inet/ntp.client and copy it to /etc/inet/ntp.conf. The default client configuration it’s just a one-liner:

multicastclient 224.0.0.1

This configuration, as explained in the same file, it’s a passive configuration for a host that just listens for NTP server putting packets on the NTP multicast network, 224.0.0.1. Obviously, if your machine it’s in a LAN without an NTP server, you’re probably never going to receive such a packet, and you should use some public NTP server instead. 

Using a server from a pool
I personally recommend using random servers from an NTP pool such as pool.ntp.org. In the official website of the NTP Pool Projectyou can find instruction about using the pool or picking up some server from the list they maintain. Pools maintained by the NTP Pool Project are organized in geographical hierarchy so that, for example, you can use server from a continent-level pool or, where available, from a country-level pool. The recipe is always the same: the nearer, the quicker, the better. In my case, I’m using the European pooleurope.pool.ntp.org and my configuration file contains:

server 0.europe.pool.ntp.org
server 1.europe.pool.ntp.org
server 2.europe.pool.ntp.org
server 3.europe.pool.ntp.org

Setting up the drift file
The only thing left to set up it’s just the drift file location, which in my case it’s:

driftfile /var/ntp/ntp.drift

Starting the service
Once your ntp.conf is set up, you can start (or restart) the ntp service:

# svcadm restart ntp
# svcs ntp
STATE STIME FMRI online 0:43:33 svc:/network/ntp:default

Querying the service
Once the service is running, you can check which server you’re using with ntpq:

# ntpq -p
remote refid st t when poll reach delay offset disp

==============================================================================
NTP.MCAST.NET 0.0.0.0 16 u - 64 0 0.00 0.000 16000.0
 +fnutt.net
Time2.Stupi.SE 2 u 45 64 377 76.74 10.285 0.73
 -sip1.viatel.ee
ntp.eenet.ee 3 u 430 1024 377 79.47 -0.988 1.05
*ntp1.net.edu.ie
ntp0.esat.net 2 u 28 64 377 63.45 4.104 0.78
+ns.airbites.bg
ntp2.gbg.netnod 2 u 27 64 377 85.13 1.723 1.05

You’ll get a similar output. After a while, your query will output similar results. The server prefixed with an asterisk is the server you’re synchronizing with. If you don’t get an asterisk after a while, probably no NTP server is reachable, which is probably due to a firewall which is blocking UDP port 123. The difference between your clock and the data provided by NTP servers can be examined by catting the drift file:

# cat /var/ntp/ntp.drift -50.645

Setting up an NTP server
Now that you have an NTP client running, you’ll probably want to setup all of your machines. If you’re in a LAN, you can setup an internal NTP server which will provide data to other clients on your LAN. As before, you can take inspiration from the server configuration file shipped with Solaris 10, /etc/inet/ntp.server. After setting up the drift file and the clients you’re going to use, you can examine the other options and fine-tune them at your taste. Let’s give a quick look at it.

server 127.127.XType.0

This line sets up the server type and the XType value must be substituted with the correct value from the provided table:

# XType Device RefID Description

# ——————————————————-
# 1 local LCL Undisciplined Local Clock
# 2 trak GPS TRAK 8820 GPS Receiver
# 3 pst WWV PSTI/Traconex WWV/WWVH Receiver
# 4 wwvb WWVB Spectracom WWVB Receiver
# 5 true TRUE TrueTime GPS/GOES Receivers
# 6 irig IRIG IRIG Audio Decoder
# 7 chu CHU Scratchbuilt CHU Receiver
# 8 parse —- Generic Reference Clock Driver
# 9 mx4200 GPS Magnavox MX4200 GPS Receiver
# 10 as2201 GPS Austron 2201A GPS Receiver
# 11 arbiter GPS Arbiter 1088A/B GPS Receiver
# 12 tpro IRIG KSI/Odetics TPRO/S IRIG Interface
# 13 leitch ATOM Leitch CSD 5300 Master Clock Controller
# 15 * * TrueTime GPS/TM-TMD Receiver
# 17 datum DATM Datum Precision Time System
# 18 acts ACTS NIST Automated Computer Time Service
# 19 heath WWV Heath WWV/WWVH Receiver
# 20 nmea GPS Generic NMEA GPS Receiver
# 22 atom PPS PPS Clock Discipline
# 23 ptb TPTB PTB Automated Computer Time Service
# 24 usno USNO USNO Modem Time Service
# 25 * * TrueTime generic receivers
# 26 hpgps GPS Hewlett Packard 58503A GPS Receiver
# 27 arc MSFa Arcron MSF Receiver

In my case, it’s just a (very) plain 1: an undiscilplined local clock.

broadcast 224.0.1.1 ttl 4

This line is the server equivalent of the multicast line seen in the default client configuration: it tells the NTP server to broadcast on the NTP multicast network.

Further readings
Complete documentation about ntp.conf syntax can be found on the xntpd man page:

# man xntpd

Setup Mysql Replication Between Linux(master) & Windows XP(Slave)

Step 1 - Configure the Master Server


First we have to edit /etc/mysql/my.cnf. We have to enable networking for MySQL, and MySQL should listen on all IP addresses, therefore we comment out these lines (if existant):


#skip-networking


#bind-address            = 127.0.0.1


 Furthermore we have to tell MySQL for which database it should write logs (these logs are used by the slave to see what has changed on the master), 


which log file it should use, and we have to specify that this MySQL server is the master. We want to replicate the database exampledb, so 


we put the following lines into/etc/mysql/my.cnf:


server-id               = 1


log_bin                 = /var/log/mysql/mysql-bin.log


binlog_do_db            = exampledb


 


 Then we restart MySQL:


/etc/init.d/mysql restart


Then we log into the MySQL database as root and create a user with replication privileges:


 


mysql -u root -p


Enter password:


 


Now we are on the MySQL shell.


mysql>GRANT REPLICATION SLAVE ON *.* TO ’slave_user’@'%’ IDENTIFIED BY ‘<some_password>’; (Replace<some_password> with a real password!) 


mysql>FLUSH PRIVILEGES;


 


Next (still on the MySQL shell) do this:


 mysql>USE exampledb;


mysql>FLUSH TABLES WITH READ LOCK;


mysql>SHOW MASTER STATUS;


 


The last command will show something like this:


 +—————+———-+————–+——————+


| File          | Position | Binlog_do_db | Binlog_ignore_db |


+—————+———-+————–+——————+


| mysql-bin.006 | 183      | exampledb    |                  |


+—————+———-+————–+——————+


1 row in set (0.00 sec)


 


Write down this information, we will need it later on the slave!


Then leave the MySQL shell:


 


mysql>quit;


 


Step 2 - Configure the Slave Server(Windows XP)





Edit the c:\program files\mysql\mysql server 5.0\my.ini 


server-id=2



master-host=db01.yourdomain.net (or IP address)
master-port=3306
master-user=slave_user
master-password=password

Step 3 - Restart Mysql Service 


goto> Control Panel>Administrative Tools>Services>Mysql 


Restart Service


mysql > Stop slave;


mysql>CHANGE MASTER TO MASTER_HOST=’192.168.10.175′, MASTER_USER=’slave_user’, MASTER_PASSWORD=’password’,MASTER_LOG_FILE=’mysql-bin.000008′,MASTER_LOG_POS=98;


mysql > Start slave;

Change Shell for user in Unix or Linux box

chsh -s /path/to/shell username


To change shell to ’sh’ from ‘bash’ for user ‘temp’  use below command:


Example : #chsh -s /bin/sh temp


 

Create a Encrypted Folder On Ubuntu for security

Step 1 : Install eCryptfs in Ubuntu

apt-get  install ecryptfs-utils

Step 2 : Create a new directory to encrypt.

#mkdir foldername

#chmod 700 foldername

Step 3 : Mount the ecryptfs to the other folder using the following command

#mount -t ecryptfs foldername[entryptedfolder] folder2[ directorywhereyouwanttomount]

It will then prompt you to answer a few questions.

Press “1″ to select passphrase

Receiving the Nagios Notifications Via Jabber

Basically the nagios calls a script written in Perl that connect to the Jabber server and sends a notice to you. 
Downloading and configuring the script 
Download the script, we should also make it executable and make sure that nagios is the owner with the commands below: 

  1. Cd / usr / local / nagios / libexec 

  2. Wget http://gnusys.net/downloads/notify_via_jabber.pl

  3. Chown nagios.nagios notify_via_jabber.pl 

  4. Chmod 755 notify_via_jabber.pl 


You must install a Perl library so that it can interact with the jabber: 

  1. Aptitude install libnet-jabber-perl 


Okay, we already have the script, now we need to make some changes so that it can connect to our server Jabber. 

Here we see what must be changed, look for this section and put the server name, port connection, the user who send the messages and password. 
use constant SERVER => ‘jabberserver.net’; 
use constant PORT => 5222; 
use constant USER => ‘user@jabberserver.net’; 
use constant PASSWORD => ‘password’; 

 
Configuring nagios 
We have to define the commands related to the hosts to be used by nagios to get him to send us messages. You can check your file and see something like commands.cfg notify-host-by-email, similar to that we need to define a Jabber. 

Edit the file commands.cfg (usually in /us / local/nagios/etc/ objects) and add the following section: 

  1. This command is used to notify recipients of service problems: 

  2. ‘Notify-by-jabber’ command definition 


define command ( 
command_name notify-by-jabber 
command_line / usr / local / nagios / libexec / notify_via_jabber.pl CONTACTADDRESS1 $ $ “$ $HostName / $SERVICEDESC is$ $ $ SERVICESTATE \ r \ nAdditional Info: SERVICEOUTPUT $ $” 


  1. This command is used to notify recipients host of problems: 

  2. ‘Host-notify-by-jabber’ command definition 


define command ( 
command_name host-notify-by-jabber 
command_line / usr / local / nagios / libexec / notify_via_jabber.pl CONTACTADDRESS1 $ $ “$ $ NOTIFICATIONTYPE: $ HostName $ HOSTSTATE is $ $ \ n $ $ HOSTOUTPUT” 


As you can see, was a program called => / usr / local / nagios / libexec / notify_via_jabber.pl, which is our script in Perl. 

Now we use these commands somewhere. Edit the file contacts.cfg (usually in / usr / local / nagios / etc / objects) and add the following section: 

Note: Set your contact information such as name, alias, email and address1, where messages are sent. 
define contact ( 
contact_name CONTACT 
use generic-contact 
Contact alias 
email user@jabberserver.net
address1 user@jabberserver.net 

host_notification_commands notify-host-by-email, host-notify-by-jabber 
service_notification_commands notify-by-jabber 
host_notification_period 24×7 
service_notification_period 24×7 
service_notification_options c, r 
)

Mysql Slave Server(Replication) Status Check Script

Below script is returning the Mysql Slave server Status.

If Replication  is running then it will not show any message but if replication is stop then it will send the email notification for status.

—————————

#!/usr/bin/env bash

repeat_alert_interval=15 # minutes

lock_file=/tmp/slave_alert.lck

active=yes

 

## Check if alert is already sent ## 

 

function check_alert_lock () {

    if [ -f $lock_file ] ; then

        current_file=`find $lock_file -cmin -$repeat_alert_interval`

        if [ -n "$current_file" ] ; then

            # echo “Current lock file found”

            return 1

        else

            # echo “Expired lock file found”

            return 2 

        fi

    else

    return 0

    fi

}

 

## Find the location of the mysql.sock file ##

 

function check_for_socket () {

        if [ -z $socket ] ; then

                if [ -S /var/lib/mysql/mysql.sock ] ; then

                        socket=/var/lib/mysql/mysql.sock

                elif [ -S /tmp/mysql.sock ] ; then

                        socket=/tmp/mysql.sock

                else

                        ps_socket=`netstat -ln | egrep “mysql(d)?\.sock” | awk ‘{ print $9 }’`

                        if [ "$ps_socket" ] ; then

                        socket=$ps_socket

                        fi

                fi

        fi

        if [ -S "$socket" ] ; then

                echo UP > /dev/null

        else

                echo “No valid socket file “$socket” found!”

                echo “mysqld is not running or it is installed in a custom location”

                echo “Please set the $socket variable at the top of this script.”

                exit 1

        fi

}

 

 

check_for_socket

 

Slave_IO_Running=`mysql -u username -p’password’ -h Slavehostip -Bse “show slave status\G” | grep Slave_IO_Running | awk ‘{ print $2 }’`

Slave_SQL_Running=`mysql -u username -p’password’ -h Slavehostip -Bse “show slave status\G” | grep Slave_SQL_Running | awk ‘{ print $2 }’`

Last_error=`mysql -u username -p’password’ -h Slavehostip -Bse “show slave status\G” | grep Last_error | awk -F \: ‘{ print $2 }’`

 

 

if [ -z $Slave_IO_Running -o -z $Slave_SQL_Running ] ; then

        echo “Replication is not configured or you do not have the required access to MySQL” | mail -s “Mysql Replication Status ” username@domain.com

        exit

fi

 

if [ $Slave_IO_Running == 'Yes' ] && [ $Slave_SQL_Running == 'Yes' ] ; then 

    if [ -f $lock_file ] ; then

        rm $lock_file

        echo “Replication slave is running”

        echo “Removed Alert Lock”

    fi

    exit 0

elif [ $Slave_SQL_Running == 'No' ] ; then

    if [ $active == 'yes' ] ; then

        check_alert_lock

        if [ $? = 1 ] ; then

            ## Current Lock ##

            echo “up” > /dev/null

        else

            ## Stale/No Lock ##

             touch $lock_file

            echo “SQL thread not running on server `hostname -s`!” | mail -s “This is subject” mail_ID@somedomain.com

            echo “Last Error:” $Last_error

        fi

    fi

    exit 1

elif [ $Slave_IO_Running == 'No' ] ; then

        if [ $active == 'yes' ] ; then

                check_alert_lock

                if [ $? = 1 ] ; then

                        ## Current Lock ##

            echo “up” > /dev/null

                else

                        ## Stale/No Lock ##

                        touch $lock_file

                        echo “LOG IO thread not running on server `hostname -s`!”

                        echo “Last Error:” $Last_error

                fi

    fi

    exit 1

else 

        if [ $active == 'yes' ] ; then

                check_alert_lock

                if [ $? = 1 ] ; then

                        ## Current Lock ##

            echo “up” > /dev/null

                else

                        ## Stale/No Lock ##

                        touch $lock_file

            echo “Unexpected Error!”

            echo “Check Your permissions!”

                fi

        fi

    exit 2

fi

Basic MySQL Commands

To login (from unix shell) use -h only if needed.


#mysql -h hostname -u root -p


Create a database on the sql server.


mysql> create database [databasename];


List all databases on the sql server.


mysql> show databases;


Switch to a database.


mysql> use [db name];


To see all the tables in the db.


mysql> show tables;


To see database’s field formats.


mysql> describe [table name];


To delete a database.


mysql> drop database [database name];


To delete a table.


mysql> drop table [table name];


Show all data in a table.


mysql> SELECT * FROM [table name];


Creating a new user.


# mysql -u root -p


mysql> use mysql;


mysql> INSERT  INTO user (Host,User,Password)  VALUES('%','username',PASSWORD('password'));


mysql> flush privileges;


Change a users password from unix shell.









#mysqladmin -u username -h hostname  -p password 'new-password'



Change a users password from MySQL prompt. 




# mysql -u root -p


mysql> SET PASSWORD FOR  'user'@'hostname' = PASSWORD('password');


mysql> flush privileges;



Recover a MySQL root password.








# /etc/init.d/mysql stop


# mysqld_safe --skip-grant-tables  &


# mysql -u root


mysql> use mysql;


mysql> update user set  password=PASSWORD("newrootpassword") where User='root';


mysql> flush  privileges;


mysql> quit


# /etc/init.d/mysql stop


# /etc/init.d/mysql  start


 


Update a root password.


# mysqladmin -u root -p oldpassword newpassword


Allow the user “user1” to connect to the server from localhost 











# mysql -u root -p


mysql> use mysql;


mysql> grant  usage on *.* to user1@localhost identified by 'password';


mysql> flush  privileges;


Give user privilages for a database.











mysql> grant all privileges on  databasename.* to username@localhost;


mysql> flush privileges;


          or


# mysql -u root -p


mysql> use mysql;


mysql> INSERT  INTO user  (Host,Db,User,Select_priv,Insert_priv,Update_priv,Delete_priv,Create_priv,Drop_priv)  VALUES ('%','databasename','username','Y','Y','Y','Y','Y','N');


mysql>  flush privileges;.


Load a CSV file into a table.


mysql> LOAD DATA INFILE '/tmp/filename.csv' replace INTO  TABLE [table name] FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'  (field1,field2,field3);


Dump all databases for backup.











#mysqldump -u root -ppassword --opt  >/tmp/alldatabases_backup.sql


Dump one database for backup.











#mysqldump -u username -ppassword --databases  databasename >/tmp/databasename.sql


Dump a table from a database.











mysqldump -c -u username -ppassword  databasename tablename > /tmp/tablename.sql


Restore database (or database table) from backup.











mysql -u username -ppassword databasename  < /tmp/databasename.sql



ALTER TABLE syntax - MySQL

ALTER [IGNORE] TABLE tbl_name alter_spec [, alter_spec ...]


 


alter_specification:


        ADD [COLUMN] create_definition [FIRST | AFTER column_name ]


  or    ADD INDEX [index_name] (index_col_name,...)


  or    ADD PRIMARY KEY (index_col_name,...)


  or    ADD UNIQUE [index_name] (index_col_name,...)


  or    ALTER [COLUMN] col_name {SET DEFAULT literal | DROP DEFAULT}


  or    CHANGE [COLUMN] old_col_name create_definition


  or    MODIFY [COLUMN] create_definition


  or    DROP [COLUMN] col_name


  or    DROP PRIMARY KEY


  or    DROP INDEX index_name


  or    RENAME [AS] new_tbl_name


  or    table_options


ALTER TABLE allows you to change the structure of an existing table. For example, you can add or delete columns, create or destroy indexes, change the type of existing columns, or rename columns or the table itself. You can also change the comment for the table and type of the table

Import CSV file directly into MySQL

You can then import it into a MySQL table by running:


 


#load data local infile 'file.csv' into table tablename


fields terminated by ','


enclosed by '"'


lines terminated by '\n'


(field1, field2, field3)

How to Grant Privileges to Users in MySQL

MySQL stores all its username and password data in a special database named mysql. You can add users to this database and specify the databases to which they will have access with the grant command, which has the syntax.

sql> grant all privileges on database.* to username@”servername” identified by ‘password’;


The next step is to write the privilege changes to the mysql.sql database using the flush privileges command.

sql> flush privileges;

How to take Linux backups powered by Rsync

RSync backups data and does it very clean and well. Rsync only transfers those data that have been modified and changed so that the destination host has an exact replica from the source host. Rysnc is a command line backup tool that handles data transfers in an effective and secure manner like any other known commercial backup softwares around. Rync blends in and integrates flawlessly with linux shell commands combined linux I/O redirections.

Here’s an altenative approach based from recent entry on creating data backups from simple one to an enterprise backup data sets using rsync.

Man Rsync:

Rsync uses a reliable algorithm to bring remote and host files into sync very quickly. Rsync is fast because it just sends the differences in the files over the network instead of sending the complete files. Rsync is often used as a very powerful mirroring process or just as a more capable replacement for the rcp command. The rsync remote-update protocol allows rsync to transfer just the differences between two sets of files across the network connection, using an efficient checksum-search algorithm.

INSTALLATION:
~~~~~~~~~~~~~~~~


Rsync installation is installed by default System Tool group installation. If rsync is not available from command line and from rpm database, you can install from yum repo using yum like so

# yum -y install rsync

To verify that rsync has been installed successfully

# rpm -qa rsync

There are two different approach on establishing rsync communication between two hosts.

First is by using a remote-shell program such as ssh or rsh. If you want to use this approach, it is required that openssh server or an active ssh connection is present from both sending and receiving host. Openssh installation and configuration would not be covered from this entry. This entry would assume that ssh daemon service is currently configured and listening properly to assigned host port.

Secondly, is by using rsync listening INETD service directly.

This entry hope to cover both of them two worlds.

Rsync usage from command line terminal
======================================


Transfer and/or update files from local host into a remote host using rsync
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


Basic rsync argument is to specify file glob, a source and a destination folder. Destination can be a local host or a remote host. A basic example to transfer from local to remote would be

# rsync -t *.mp3 remoteusername@remotehost:remote_destination_folder

The above would transfer, using rsync, all *.mp3 files from current local directory into remote_destination_folder of remotehost using remoteusername for authentication. A destination host can be locally or remote host and can be specified as hostname or IP address.

Legend:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-t preserve time of selected files
*.mp3 selected files to be transferred
remoteusername bash enabled user account from destination host
remotehost destination host where remoteusername is allowed to have access
remote_destination_folder destination folder from destination host owned and accessible
by remoteusername
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


For recursive traversing of directory folders for transferring data from current host to another host using rsync, this would be like so

# rsync -avrzt /var/www myuser@server1:/var/backup/
Legends:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-a archived mode enabled
-v verbosed mode enabled
-t preserve time stamp
-z compression transfer enabled
-r resursive mode enabled
/var/www selected folder or files glob source location
myuser user account from destination host
server1 destination host
/var/backup destination folder
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


The above command would issues rsync with verbose mode enabled, compression enabled, arhived mode of rsync data transfer/update. Rsync transfers files and folders from /var/www (including the www folder) into /var/backup folder of server1 host using myuser as login credentials. The /var/backup from destination host is owned or writeable by myuser . All rsync files are created with file ownership and permission owned by myuser having 600 file mode. The transfer would be done preserving symbolic links, devices, attributes, permissions and ownerships of files.

# rsync -avz /var/www/ myuser@server1:/var/backup/www

Appending / from selected file glob like the above command issues the same rsync command with same argument. The only difference is that source folder is not created .

Transfer files from remote host to local host using rsync
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


# rsync -avrzt remoteusername@remotehost:remote_source_folder destination_folder
# rsync -avrz myuser@server1:/var/backup/www /var/www/


To transfer multiple files with different file extension from local host to remote host using rsync with file glob would be

# rsync -avrz file1.mp3 *.doc myuser@server1:/home/folder/location

To transfer multiple files from multiple directory sources using rsync would be

# rsync -avz `find /etc -name *.conf` user@remotehost:/user/folder

Linux command line tools when combined with each other creates another set of power tools. The above command searches all *.conf files from /etc folder and transfer them to /usr/folder of remotehost via rsync and using user the user login name. This can be handy if you like to backup specific pattern of files from your system or user accounts individual address books, files like that.

To transfer multiple files from multiple directory with some file excemptions using rsync and grep from local to remote would be like so:

# rsync -avz `find /etc -name *.conf | grep -v yum.conf` user@remotehost:/user/folder

Alternatively,

# rsync -avz $(find /etc -name *.conf | grep -v yum.conf) user@remotehost:/user/folder

and for multiple file rsync with multiple file from multiple source location with multiple file transfer exceptions would be like so

# rsync -avz `find /etc -name *.conf | grep -v ‘yum.conf\|xorg.conf’` user@remotehost:/user/folder

To transfer of all your back up files ending in .tar file extension from any location would be like so

# rsync -avz `find / -name *.tar` user@remotehost:/user/folder

You will noticed that rsync can transfer data at high speed rate using rsync algorithm specially if those files and folders to be transferred are existing already from remote host.

To transfer multiple file(s) with multiple exclusions using rsync would be like so:

# rsync -avz * user@remotehost:/location –exclude=*.php –exclude=*.mp3

To transfer files in batch mode based from file lists using rsync would be

Assuming listing.txt is created with contents like below

# cat listing.txt
~~~~~~~~~~~~~~~~~~~~~
files0123.txt
files0124.txt
files6124.txt
files6126.txt

snipped

files32126.txt
~~~~~~~~~~~~~~~~~~~~~


and feeding the above file to rsync as batch mode input like so

# rsync -avzt –files-from=listing.txt user@remotehost

:/destination/

To rsync multiple folder source location using rsync would be like so

# rsync -avz –files-from=/home1 /home2 user@remotehost:/destination/

If both location contains abc.txt, the latest abc.txt would be transferred to remote host. If /home1/www exist and /home2/www exist, the files from both source kicatuib would be merged into /destination/www .

Fire up two new terminal windows, and from the first window, establish ssh connection from local to remote rsync destination. Then from the second windows try to issue these rsync command. You will notice that rsync never ask any password any more since there is an existing ssh connection with local host to remote host.

More rsync arguments
====================


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-W transfer the whole files without considering any update changes from existing destination file or folder
–progress show progress bar and/or percentage
-4 prefers IPv4
-6 prefers IPv6
–bwlimit=KBPS execute rsync with bandwidth limit rate in KBPS
-h display a more human readable screen output
–log-file=FILE dumps file activity into a file
–ignore-existing ignores already existing copy from destination host
–max-size tells rsync to avoid transferring files larger than specified size like
–max-size=10m avoid file transfer with 10MB in filesize
–port tells rsync to connect to rsync server with a different port, default is 873
–stats tells more file transfer info and rsync algorithm stats on the fly
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


At this point, be reminded to kick the black ninja box to refresh possible monitor related eye strains.

Using Rsync in Daemon Xinetd Mode
=================================


Edit /etc/xinetd.d/rsync and modify

disable = yes
to
disable = no


To run rsync in deamon mode via xinetd, make sure you have similar lines from your /etc/xinetd.d/rsync file like shown below:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
service rsync
{
disable = no
socket_type = stream
wait = no
user = root
server = /usr/bin/rsync
server_args = –daemon –config=/etc/rsync/rsyncd.conf -v
log_on_failure += USERID
}
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


Create /etc/rsync/rsyncd.conf as a default conf file for rsync in daemon xinetd mode.

Sample rsyncd.conf file
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
uid = rsync-user
gid = rsync-user
use chroot = yes
read only = yes
pid file = /var/run/rsyncd.pid


# access list, edit the IP network block to suit your needs
hosts allow=192.168.0.0/255.255.0.0 192.168.1.0/255.255.255.0
# deny anything else
hosts deny=*


# limit connections
max connections = 5
#greeting file
motd file = /etc/rsync/rsyncd.motd
#log file
log file = /var/log/rsync.log


#rsync shared folder
[myrsync]
#make your UID/GUID above owns the below folder and files
#all files and folder would be seen by rsync client
path=/home/rsync-user/rsync-folder
comment = Linux Rsync Server
exclude = *.mp3


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Create rsync greeting MOTD file like so

# cat /etc/rsync/rsyncd.motd
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Welcome to my Rsync Server!
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


Restart xinetd service for the changes to take effect like so

# service xinetd restart

Verify that rsync is running as xinetd daemon service mode using one tool like ss

# ss -a | grep rsync
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LISTEN 0 0 *:rsync *:*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


Rsync uses port 873 as its default port for xinetd service. Make sure it is also open from your current firewall settings. A sample firewall rule for opeing rsync from your firewall would be like so

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-A RH-Firewall-1-INPUT -m state –state NEW -m tcp -p tcp –dport 873 -j ACCEPT
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# service iptables restart


If you wish to have a passwordless rsync, you need to refer to passwordless/passphraseless ssh from one of last month’s entry.

Rsync in daemon mode via command line
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


An alternative to run rsync in daemon mode is by specifying it from command line. Like so

# rsync –daemon –address host-IP-address –config=/etc/rsync/rsyncd.conf -v –port=873

Legend:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
–daemon enables rsync to be run as daemon mode from terminal
–port specifies port number to use
–config specified rsync conf file
host-IP-address IP address where to bind rsync from
-v enables verbose mode
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


Listing out files and folder from rsync server
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


To list out the files from rsync server, simply point your rsync client to the host running rsync in daemon mode or server mode. To test your rsync server from a client linux host would be like so

# rsync host-IP-address::

Alternatively,

# rsync rsync://host-IP-address/myrsync
# rsync rsync://host-IP-address/myrsync/folder1


From the above rsync command, you should be seeing files from /home/rsync-user/rsync-folder folder from rsync server you defined from /etc/rsync/rsyncd.conf.

Syncing File and Folders from Rsync Server
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


To sync and download files from rsync server.

# rsync rsync://host-IP-address/myrsync/folder1 .

The above command downloads files including the whole folder1 folder from rsync server and saves it to current directory

# rsync rsync://host-IP-address/myrsync/folder1/ .

The above command downloads only files from folder1 to current folder location

Rsync Log Monitoring
~~~~~~~~~~~~~~~~~~~~


Monitoring rsync server messages for any errors or system messages would be like so

# tail -f /var/log/rsync.log

Final Note:

Rsync can also be used for source code control and management for delivering a centralized and distributed sync source codes from rsync server to group of programmers or source developers among departments, more like a CVS approach.

Using rsync linux command provides many abilities and benefits. To name a few, mirror an entire harddisk or partitions, folders and files, entire domain websites, mail spools for replica servers, user’s home folder, a mirror FTP site and much more.

Backup like the enterprise way, use RSync.

sanjay's shared items

My Blog List