Showing posts with label About Server Load. Show all posts
Showing posts with label About Server Load. Show all posts

Sunday, May 25, 2008

Nobody Prevention Script

A big problem today is the abundance of Spammers listing as "Nobody".

This tutorial will outline how to properly set it up on Cpanel based systems.

root@yourserver [~]# mv /usr/sbin/sendmail /usr/sbin/sendmail.real
// backup your existing sendmail in the event of an error.

root@yourserver [~]# pico /usr/sbin/sendmail
// Open this badboy up, now paste the code below into it.

Code:

#!/usr/local/bin/perl

# use strict;
use Env;
my $date = `date`;
chomp $date;
open (INFO, ">>/var/log/formmail.log") || die "Failed to open file ::$!";
my $uid = $>;
my @info = getpwuid($uid);
if($REMOTE_ADDR) {
print INFO "$date - $REMOTE_ADDR ran $SCRIPT_NAME at $SERVER_NAME \n";
}
else {

print INFO "$date - $PWD - @info\n";

}
my $mailprog = '/usr/sbin/sendmail.real';
foreach (@ARGV) {
$arg="$arg" . " $_";
}

open (MAIL,"|$mailprog $arg") || die "cannot open $mailprog: $!\n";
while ( ) {
print MAIL;
}
close (INFO);
close (MAIL);

Now save the work above...

root@yourserver [~]# chmod +x /usr/sbin/sendmail
// make sendmail executable

root@yourserver [~]# echo > /var/log/formmail.log

root@yourserver [~]# chmod 755 /var/log/formmail.log (If no work change to chmod 777)

root@yourserver [~]# pico /var/log/formmail.log
// above is where all the data is stored, it may take up to a couple hours for it to begin filling with data. However if you notice an extreme amount of instances of a user sending mail, it may be spam. In the past without this script it would list them as "Nobody". With the script in place it lists the user and where the mail is being setn from.
--------------------------------------------------------------------------
As always it's a good idea to routinely check who and how many perl processes are running as this is also another large gateway for outgoing spam.

root@yourserver [~]# ps aux | grep perl
// if you see a user with one to many perl processes running, and nothing much going on at his site, in most cases that i've seen this is the source of spam.
__________________

Saturday, April 12, 2008

How to set limit to remove the Frozen Maiils Automatically ?

vi /etc/exim.conf

timeout_frozen_after = 8d ( 8 Days )

/scripts/restartsrv_exim

How to trace the DDOS attack on the server ?

1. Your should have following setting at the time of DDOS attack in httpd.conf:

TimeOut = 20

KeepAlive Off

MaxClients 384

MinSpareServers 20

MaxSpareServers 25

2. in /usr/local/ddos/ddos.conf

NO_OF_CONNECTIONS=20

3. You should have 7 SSH session and 1 WHM at the time of DDOS.

4. Check the domlogs to trace out a particular website for the DDOS. Use the following command to check the latest updated domlog file for the website.

ll -lt |less

5. You should keep the following command to check the DDOS.

top, access_logs, error_logs,

ps aux | grep php

check the apache status in the WHM

cd /root/nobody_check

./apachetrace

cd /tmp

ls and check suspected scripts.

6. If necessary reboot the server. This will kill the http process which is causing the DOS. When the server is up, that process will start again and at that time you can trace it and kill it.

7. You can change the permission of the suspected domain. Make the DNS changes to 127.0.0.1 . please use TTL 20 for fast dns propagation. Don't set the redirectio for the website in httpd.conf.

8. After the DDOS attacker is trace down, don't forget to revert back the changes make to https.conf.

9. Suspend the suspected domain and mail the client about this.

check queue by whm

if queue is high I will check queue by whm

I will scroll bottom I will see which domain is most of time or aol

If I find any I will click on id I will see his email headers so we ill get spammer.

I know all mails are not spamer in such case we will delete mails. Only

Tail –f /var/log/exim_mainlog |grep sendmail

Tail –f /var/log/exim_mainlog |grep tmp

Tail –f /var/log/exim_mainlog |grep public_html

Tail –f /var/log/exim_mainlog |grep hostname


Or

Cd /var/spool/cron

Check if someone is sending mail or bulk list.

2)check aol mails

if we find continuously mails from AOL

logged at server

check mail queue who is sending mail to AOL id check header and suspend the account.

As per apache load.

Ps auw |grep nobody

We check is there any old or bad process kill it

We check apache status.

We check access_logs

We check domlogs of domain

For control

1)cap domain name for ip limit

2)if ddos 127.0.0.1 update client

3)if accessing only one file redirect it

4)if ~access ban by mod_sec

in ddos we used following method

In DDOS ON mode usually the best setting is

/usr/local/ddos/ddos.conf

NO_OF_CONNECTIONS=100

/etc/httpd/conf/httpd.conf

Timeout =10

KeepAlive = OFF

mod_evasive = uncomment

===========================================================

In DDOS OFF mode

/usr/local/ddos/ddos.conf

NO_OF_CONNECTIONS=650

/etc/httpd/conf/httpd.conf

Timeout =50

KeepAlive = On

mod_evasive = comment

Ddos mode should be off when attack stop.

We check proc/pid too

mysqladmin processlist

if one database again and again we suspend him.

Friday, April 11, 2008

Script to catch malicious perl scipts uploaded to /tmp

Following script can help you to catch malicious perl scipts uploaded to /tmp.

#!/bin/bash
#Script designed for http://dantechie.blogspot.com/. Its GNU,FREE,DISTRIBUTABLE

ROOTBADSCRIPTS=/root/badperlscripts
EMAILLOG=/root/emailperlscriptlog
LOAD=`cat /proc/loadavg | awk ‘{print $1, $2, $3}’`
HOST=`hostname`
TIME=`date`
ADMINEMAILS=”,admin@admin.com”
if [ ! -e $ROOTBADSCRIPTS ]; then
mkdir $ROOTBADSCRIPTS
chmod 700 $ROOTBADSCRIPTS
fi
rm -f $EMAILLOG
touch $EMAILLOG
for FILES in /tmp/*
do
if [ ! -d $FILES ]; then
if [ -w $FILES ]; then
#grep perl $FILES > /dev/null 2>&1
head -1 $FILES | grep perl > /dev/null 2>&1
if [ $? -eq 0 ];
then
echo “$TIME $HOST $FILES — perl script found Load : $LOAD ” >> $EMAILLOG
mv -f $FILES $ROOTBADSCRIPTS
killall -9 perl > /dev/null 2>&1
killall -9 perl > /dev/null 2>&1
killall -9 perl > /dev/null 2>&1
#//chown root.root $FILES
#//chmod 000 $FILES
#//chattr +i $FILES
fi
fi
fi
done
for FILES in /dev/shm/*
do
if [ ! -d $FILES ]; then
if [ -w $FILES ]; then
#grep perl $FILES > /dev/null 2>&1
head -1 $FILES | grep perl > /dev/null 2>&1
if [ $? -eq 0 ];
then
echo “$HOST $FILES — perl script found Load : $LOAD ” >> $EMAILLOG
mv -f $FILES $ROOTBADSCRIPTS
killall -9 perl > /dev/null 2>&1
killall -9 perl > /dev/null 2>&1
killall -9 perl > /dev/null 2>&1
#//chown root.root $FILES
#//chmod 000 $FILES
#//chattr +i $FILES
fi
fi
fi
done
if [ -s $EMAILLOG ]; then
cat $EMAILLOG | mail -s “Perl Script report on $HOST” $ADMINEMAILS
fi
exit

Friday, March 21, 2008

SERVER LOAD TWEAKS...APACHE/MYSQL/SMTP-POP3/EXIM/IMAP

Q:- How to trace the server load? (Inludes with all application.)


The steps are according to services :-

----------------------------------------------------------------

1. Apache :-

  • TOP command >> check for many httpd processes
  • Login to WHM >> Check for "Apache ststus",if find anyone downloading mp3,rar,exe,zip files then suspen that account
  • Also used to check "cpu/memory/Mysql Usage" option from WHM >> Here we can find actual CPU and Memory usage for particular domain.


----------------------------------------------------------------

2. Mysql :-

  • TOP command >> check for many httpd processes
  • MySQLadmin process / mysqladmin status
  • Also used to check "cpu/memory/Mysql Usage" option from WHM >> Here we can find actual CPU and Memory usage for particular domain.


----------------------------------------------------------------

3. Ftp :-

  • ps -aux | grep ftp >> used for checking ftp action taken by user like uploading /downloading files
  • tail -f /var/log/secure


----------------------------------------------------------------

4. SMTP / POP3 /IMAP :-

  • tail -f /var/log/exim_mainlog >> checking for logs and check which email addresses is continuesly scrolling and confirm is it doing spaming,if confirmed then suspend account
  • tail -f /var/log/exim_mainlog | grep public_html >> check for spamming if anybody is using php script for sending mail
  • Login to whm and select "Manage Mailqueue" to find the email address which is doing spamming.

----------------------------------------------------------------

First of all check the load and if its above safe limits, we have to settle it down. run deep scripts untill you get the messages "no processes found" for mysql, exim and http. check the load again. restart the services by proper restart scripts
s-http, s-mysql, s-exim. Then proceed for investigation.

* top, shift p, check processes taking load and are in plenty.
* if apache
* quickly go to whm and check apache status and do the needful. if theres nothing in apache
check netstat -n|less . it can be http attack.
* if mysql-
do mysqladmin process and look for processes and queries.
* if exim-
go to /var/log/exim_maillog and check the logs for spam.
this can be done by grepping the logs as follows:
tail -f /var/log/exim_mainlog|grep /tmp
tail -f /var/logs/exim_mainlog |grep public_html
tail -f /var/logs/exim_mainlog |grep sendmail
check mail queue
check for frozen mails and delete frozen mails.
* check io wait if its more.
there can be some reasons for this.
* any user may be downloading heavy files- this you can see in the apache status.
* you can see the heavy cpu consuming processes on the server by ps auxw|grep mvi,mgp,mp3,pkgacct,backup,gzip and you can get the processes. check it and kill it as necessary.
* last is check out for bad processes by ps auxw|grep nobody and kill the bad processes. to know more about what is happening behind the process you can check at /proc/procid.

----------------------------------------------------------------

  • 1> ps aux | grep nobody or gzip /backup /fixquota
  • 2> TOP / shift + p / shift + m / k =kill
  • 3> tail -f /var/log/... | grep ... .avi/.mpg/.rar/.jpg all logs
  • 4> cd /proc/pid ls -alh
  • 5> Apache status/ cpu mysql memory usage form whm
  • 6> netstat -n
  • 7> w
  • 8> Event Viewer log / Task manager for windows
  • 9>tail -f /var/log/exim_mainlog|grep tmp /sendmail /public_html


----------------------------------------------------------------

  • 1) top.....to see the process list, then accroding kill the process which is taking load
  • 2) ps -aux
  • ps -aux | grep gzip, backup, pkg
  • tail -f /var/log/exim_maillog | grep sendmail, public, tmp
  • 3) w to see whois online
  • 4) kill httpd, mysqld, cppop
  • 5) netstat
  • 6) mysqladmin process :- to see the mysql process
  • 7) /scripts/restartsrv_service name:- to restart the service if it goes down
  • 8) tail -f /etc/httpd/logs/access_log
  • tail -f /etc/httpd/logs/error_log


or you can manage the serverload using WHM

1) under the server status option you can see the
apache status...
CPU/Mysql usages/memory
service status and you can manage the server
2) SQL services under this option you can see the mysqladmin process

3) restart services:- using this option you can restart the services



----------------------------------------------------------------


---
top
---
Will display the processes that are using the maximum processor resources
We can use various options to monitor and control process through top like shift + p, shift + m and k which is used to kill processes. r can be used to renice a process and prioritise a process. In case of high i/o wait we need to check the logs for
high resources using processes.

---------
uptime
---------
It displays the the time since the server has been up and running, number of users logged in and the load average. Similarly we can use 'w'.

------------
ps -auxwf
------------
Will display the process with details like, username, pid, resource usage and child processes. It is very effective in monitoring processes.
We generally use ps -auxwf | grep gzip
ps -auxwf | grep backup
ps -auxwf | grep pkg
For bad processes - ps -auxwf | grep nobody

-------------------
Kill and Kill All
-------------------
used to kill processes or services that are found to be eating up server resources.

--------------
Spamming
--------------
To check spamming we can watch for the mail logs using :
tail -f /var/log/exim_mainlog | grep sendmail
tail -f /var/log/exim_mainlog | grep tmp
tail -f /var/log/exim_mainlog | grep public_html
as spamming can be done from a user's public_html directory using a script or through sendmail. Another way of spamming is using the tmp directory as it is the 'world writable directory'.

--------------
WebServer logs
--------------
We can check for customized logs in the WHM under the Server Status section.
We can trace the user responsible for high web server resource usage by the folowing command
tail -f /etc/httpd/logs/access_log | grep mp3
tail -f /etc/httpd/logs/access_log | grep rar
tail -f /etc/httpd/logs/access_log | grep wav etc

tail -f /etc/httpd/logs/access_log | grep 408 can be used to check for DDOS attacks on the server.

--------
mysql
--------
Apart from top and ps, 'mysqladmin processlist' can be used to check the mysql processes, users and the type of process/query being run by the user.

Killing a proceess is the first option to control server load, restarting the affected server is another option. Still if the load is high we track down the responsible user and suspend him.(This applies for all servers,i.e,apache, mysql, exim etc.)


----------------------------------------------------------------

1. top - check load average, iowait, httpd, mysql, exim etc.
2. P = CPU Usage, M = Memory Usage, K = kill unwanted processes.
3. If load is high, run "deep".
4. Login to WHM of the server and check apache, cpu/memory, mysql status.
5. If any user found downloading gif images, mp3, etc; suspend that particular user.

iowait is high, someone is backing up their files; run
# ps aux | grep pkg
# ps aux | grep gzip
# ps aux | grep backup

Spamming check-
# tail /var/log/exim_mainlog –f | grep public_html
MySQLdump check-
# ps aux | grep mysqldump
# mysqladmin processlist -- mysql status
Bad processes running-
# ps aux | grep nobody

Service restart commands-
#/scripts/restartsrv_mysql
#/scripts/restartsrv_httpd
#/scripts/restartsrv_exim


----------------------------------------------------------------



By using a top command you can find out the process which is causing the load on the server. You can use kill or kilall command to kill that process. OR you can run deep command which will kill all the httpd, exim and mysql process. once the server load comes down restart the service which you have killed.

1. Apache :-

Using top command, we will come to know whether httpd service is eating up high resources on server. if so then kill the httpd service and restart it again when load comes to normal.

You can check if any backup is going on, run the following commands:
# ps aux | grep pkg
# ps aux | grep gzip
# ps aux | grep backup
If any backup process is going on, kill that process.

Also Login to WHM and Check for "Apache ststus", if find anyone downloading mp3,rar,exe,zip files then suspend that account.

the other way to check any download is going on is by using
ps auxw | grep nobody | grep mp3 , jpeg, wmv, mpeg, rar and kill that process.


ALso you can used to check "cpu/memory/Mysql Usage" option from WHM. Here we can find actual CPU and Memory usage for particular domain.

2. Exim Mail :-

Check for spamming by checking the logs for exim

tail -f /var/log/exim_mainlog >> checking for logs and check which email addresses is continuesly scrolling and confirm is it doing spaming,if confirmed then suspend account.

tail -f /var/log/exim_mainlog | grep tmp

Login to whm and select "Manage Mailqueue" to find the email address which is doing spamming.

3. MySQL :-

Use the mysqladmin command as mysqladmin process / mysqladmin status

Also used to check "cpu/memory/Mysql Usage" option from WHM. You can get actual CPU and Memory usage for particular domain.


4. Ftp :-


ps -aux | grep ftp >> used for checking ftp action taken by user like uploading /downloading files
tail -f /var/log/secure

5. ps command :-

Use ps -auxw command ALWAYS to check if there is bad processes running by doing ps aux|grep nobody. You'll sometimes see bad scripts running as nobody. Normally only httpd, merlange chat, and sometimes proftpd are run as user nobody. so if you find any other process is running as user nobody kill that process.

kill -9 pidofproc


----------------------------------------------------------------

Some basic commands to see load average and process running on the server,
1]top :: This command is very useful for system administartion. Basically it gives you summary view of system, including number of users, memory usage, CPU usage and active processes.
Shift+p =>list all processes accourding to maximum CPU usage.
Shift+m=>list all processes accourding to maximum memory usage.

2]w :: This commands gives us information regarding who is logged into server and what processes they are running
w -s , gives you shorter process listing.

3]uptime:: It will also gives us information regarding the number of user logged into server, Current time, time since server is up, load average.

4]ps :: list the current running processes.
ps -aux , gives us information of users, PID, resoure usages like CPU and memory, processes running.

Reasons for increase of load on the server::

Load on server will get increase due to sevral reasons sated below,

1.many httpd processes
2.any user is downloading mp3, exe, zip files
3.email spamming
4.uploading/downloading files via FTP
5.mysql processes and queries run by user

Troubleshooting::

1.If many load increased suddenly, fire "deep" command, which kills httpd, mysql and exim processes.
2. Also Login to WHM and Check for "Apache ststus", if find anyone downloading mp3,rar,exe,zip files then suspend that account.
3.For spamming, you need to fire following commands,
tail -f /var/log/exim_mainlog | grep public_html



----------------------------------------------------------------
1)top
2)w
3)shift+m
4)shift+p
5)tail -f /usr/local/apache/logs/access_log
6)tail -f /usr/local/apache/logs/error_log
7)tail -f /var/log/exim_mainlog


----------------------------------------------------------------


1. Check for server load using top command with following options:
Shift p CPU Usage,
Shift m Memory Usage
& check which process is taking load with the help of above two options.
Kill the responsible process using k option.
2. Check for the downloads using
# ps auxw | grep nobody | grep mp3 , jpeg, wmv, mpeg, rar
# ps auxw | grep gzip, backup, fixquota
Suspend the perticular account who is repeatedly downloading the above mentioned files.
3. Check for access & error logs for following options
# tail -f /etc/httpd/logs/access_log | grep 408, zip
# tail -f /etc/httpd/logs/error_log | grep 203
4. Check mail spamming with following commands.
#tail -f /var/log/exim_mainlog | grep sendmail, public_html, tmp
5. Login to WHM of the server and check apache, cpu/memory, mysql status & check for frozen mails in mail queue manager.
6. Check Mysql errors with
# mysqladmin processlist
check the users, command, time & information fields.
7. If you are making changes to httpd.conf then first run # httpd -configtest before restarting httpd to reduce the downtime.
8. Restart the perticular service causing load tomgo hihg with # /scripts/restartsrv_httpd, exim, mysql

----------------------------------------------------------------


Sunday, March 16, 2008

Exploit Removal Guide

The following is a first step in finding and removing exploits and root kits on a Linux or BSD system.

1. EXECUTE THE FOLLOWING COMMANDS TO HELP PREVENT UPLOADS OF EXPLOITS:

chmod 0750 `which curl` 2>&-; chmod 0750 `which fetch` 2>&-; chmod 0750 `which wget` 2>&-

2. EXECUTE THE FOLLOWING COMMANDS TO CHECK FOR POSSIBLE EXISTING EXPLOITS:

shfor x in "/dev/shm /tmp /usr/local/apache/proxy /var/spool /var/tmp"; do ls -loAFR $x 2>&- grep -E "^$^/ apache nobody unknown www web htdocs " grep -E "^$^//$\*$\.pl$" grep -Ev "sess_" tee exploits.txt; done; echo -e "\n\nPossible Exploit Files and Directories: `grep -Ev "^$^/" exploits.txt wc -l tr -d ' '`" tee -a exploits.txtexit

Lines ending with an asterisk '*', '.pl', or a slash '/' are possible exploit files or directories which should be investigated and removed followed by rebooting the server to kill any running exploit processes. You can refer to the exploits.txt file generated by the above commands for later reference.

3. You should also install and run the progam called rkhunter.
Rootkit Hunter is scanning tool to ensure you for about 99.9% you're clean of nasty tools.
This tool scans for rootkits, backdoors and local exploits by running tests like:

- MD5/SHA1 hash compare
- Look for default files used by rootkits
- Wrong file permissions for binaries
- Look for suspected strings in LKM and KLD modules
- Look for hidden files
- Optional scan within plaintext and binary files
WWW: http://www.rootkit.nl/

On BSD sytems:
cd /usr/ports/security/rkhunter; make install clean; rehash; rkhunter -c(or for help with rkhunter arguments do: rkhunter -h)

On RedHat, Fedora, CentOS systems:
yum -y install rkhunter;
rkhunter -c
(or for help with rkhunter arguments do: rkhunter -h)

Monday, March 10, 2008

Kill nobody /user process

Kill nobody process

ps aux grep nobody awk '{print $2}' xargs kill -9
Kill user process
ps aux grep apache awk '{print $2}' xargs kill -9

Wednesday, March 5, 2008

How to block an IP/netblock using iptables

To block IPs in Linux you use a program called iptables that should already be installed on your server. To issue the neccessary commands you will need to login to your server via SSH as the root user.

Adding Temporary Rules:
To make only temporary rules that will not survive a reboot you can do the following steps. Once the rules are setup correctly they can be made permanent
Step 1) Determine which IPs need to be blocked from accessing your server.
Step 2a) To block a single IP issue the following command at the command prompt
iptables -I INPUT -s -j DROP
Step 2b) To block a range of IPs issue the following command at the command prompt. This will block all ips starting at and incrementing by one until it reaches and includes
iptables -I INPUT -s : -j DROP
Step 2c) To block a Netblock of IPs issue the following command at the command prompt. This will block all ips that fall into the subet by applying the to .
iptables -I INPUT -s / -j DROP

Removing Temporary Rules:
Step 1) At the command line type the following command to display the list of current rules:
iptables -L
Step 2) The previous command should have displayed "Chain INPUT" followed by a list of rules. The top most rule is considered to be Rule 1. Count down to the rule you wish to remove and note its number. So the first rule is Rule 1, the second is Rule 2, etc.
Step 3) Type in the following command where is the number of the rule you wish to delete
iptables -D INPUT

Making/Adding Permanant Rules:
The above rules will only last until your server is rebooted. There are two ways to make make the rules permant on a RHEL or CENTOS based system. You can setup temporary rules as shown above and then save the current configuration when you are sure all the rules are correct. To do this you type in the following command which will save the rules and make sure they run at the next reboot.
iptables-save > /etc/sysconfig/iptables; chkconfig iptables on

The second method is to add the rules manually to the file /etc/sysconfig/iptables and then restart iptables. The rules themselves remain mostly unchanged as seen below. The only difference is that you do not call the iptables command.
1a) To block a single IP add the following to /etc/sysconfig/iptables.
-I INPUT -s -j DROP
1b) To block a range of IPs add the following to /etc/sysconfig/iptables.
-I INPUT -s : -j DROP
1c) To block a Netblock of IPs add the following to /etc/sysconfig/iptables
-I INPUT -s / -j DROP
2) Restart iptables by issusing the following command:
service iptables restart
3) Ensure iptables runs at reboot
chkconfig iptables on

Removing Permanant Rules:
1) Delete the rules from the file /etc/sysconfig/iptables
2) Restart iptables by issusing the following command:
service iptables restart

How to find all IPs bound to a machine that are not hosting sites

How to find all IPs bound to a machine that are not hosting sites

This command can be used to find free IPs on a non-panel Linux server.

echo System has $(ifconfig | grep "inet addr:" | grep -v 127.0.0.1 | awk '{print $2}' | sed s/"addr:"// | sort -n > /tmp/boundips && cat /usr/local/apache/conf/httpd.conf | grep ""$// | sed s/">"$// | sort -n | uniq > /tmp/usedips && diff /tmp/boundips /tmp/usedips | sed -n /^"< "/p | sed s/^"< "// | wc -l) free IPs: && diff /tmp/boundips /tmp/usedips | sed -n /^"< "/p | sed s/^"< "// && rm -f /tmp/boundips /tmp/usedips

Note: On a cPanel server, there is a tool called "Rebuild the IP address pool", which performs the same function.

How to clean Exim's Mail queue

How to clean an Exim queue

* How to clean an Exim queue
o Using WebHost Manager
o Advanced Users
o Very Advanced Users

Using WebHost Manager

1. Login to WHM.
2. On the Main page, click on the email icon:
3. In the mail menu, select "Manage Mail Queue":
* Note: Instead of steps 2 and 3, you can select "Manage Mail Queue from the left margin:
4. Find the message you wish to delete and select "Delete":

Advanced Users
WARNING: The information below is intended for users with advanced knowledge of operating systems, control panels, and other aspects of server management. Do NOT simply copy and paste commands to resolve issues as you may severely harm your server, cause downtime, or incur billable support not covered under your DEFCON plan. Please open a support ticket if you feel uncomfortable with the suggestions provided.

There are many times when a mail queue may become filled with what is essentially junk mail. At extreme levels, this can cause high load and delayed mail delivery.
You can use a variation of the following command via a shell prompt to delete only these junk messages from the mail queue:

grep -lR KEYWORD /var/spool/exim/input/* | xargs rm -f

This will purge the mail queue of any messages that contain KEYWORD.
Any messages deleted in this manner are IRRETRIEVABLE (aka: Gone for good)
Is this dangerous? You bet it CAN be. If you have any doubts, open a support ticket.

Let's look at an example.
Imagine a case in which users are abusing a bad FormMail.pl installation on server.fastservers.net. Almost all of these messages will contain the username "nobody@server.fastservers.net". We want to delete ALL of these messages, so we would insert "nobody@server.fastservers.net" in place of "KEYWORD" in the command above.
This will delete all messages in the exim queue that contain the keyword "nobody@server.fastservers.net".

Very Advanced Users
Grep does regular expressions. This means you can catch tricky spammers with a little mind power. For example:

grep -lRP Vz*Az*Lz*Iz*Uz*M /var/spool/exim/input/* | xargs rm -f

This command will delete any message containing the letters V, A, L, I, U, M, in that order, and with 0 or more "z" characters between them. A few of the keywords it will match are listed:
VALIUM
VzALIUM
VAzLIzUzM
VALzIUM
VzALzIUM
You get the idea.

Prevent User nobody from Sending Mail

To prevent spammers from sending mail through vulnerable scripts or uploading and running their own scripts to send spam from your server, you may want to prevent the user nobody from sending mail from the server. The users on the server will still be able to have scripts that send to their local addresses. If they want the script to send to an outside address, they can create a forwarder and then have their script send mail to the forwarder.

This security feature is enabled by going to Tweak Settings in the Server Configuration section of the WHM. In the Mail section, check "Prevent the user "nobody" from sending out mail to remote addresses". Finally, scroll to the bottom and click Save.

Tracking Down Spam Scripts in cPanel

Tracking Down Spam Scripts in cPanel
If you're receiving complaints that your cPanel server is sending out spam, there is a fair chance that there is a malicious script somewhere on the server. It can be a little tricky to locate such a script, but here is a trick that may help:

Log into your WHM interface by using the root user and root password.
Click on 'Exim Configuration Editer' under 'Service Configuration'.
Click the box at the top labeled 'Switch to Advanced Mode (Edit Raw Configuration File)'.
Copy the following and paste into the top box:

log_selector = +address_rewrite +all_parents +arguments +connection_reject +delay_delivery +delivery_size +dnslist_defer +incoming_interface +incoming_port +lost_incoming_connection +queue_run +received_sender +received_recipients +retry_defer +sender_on_delivery +size_reject +skip_delivery +smtp_confirmation +smtp_connection +smtp_protocol_error +smtp_syntax_error +subject +tls_cipher +tls_peerdn

Next, scroll down to the bottom and click the 'Save' button.

At this point, you can log out of WHM, and log into your server via SSH.

Now we want to watch the mail program's log files, to see where the messages are comming from on your server. Type the following at the command line:

tail -f /var/log/exim_mainlog | grep cwd

This will show exactly what is being logged by the mail server, but will only display the relevant lines.

The output will be something like the following. 'cwd', in this case stands for 'current working directory', or the place on the server where the messages originate.

cwd=/var/spool/exim --> This one is nothing to worry about; this is normal operation.
cwd=/tmp --> This is the temporary directory. If you see entries with this, you should probably investigate further.
cwd=/home/accountname/public_html/forums/tmp --> This one is definately something that should be investigated.


ls -la /home/accountname/public_html/forums/tmp --> This will list the directory's contents.

By listing the contents of the directory that looks suspicious (in this case, /home/accountname/public_html/forums/tmp), you can discover the name of the script. To check and see if it is currently running, type the following:

ps aux | grep

This will give an output that includes a number known as the process ID. You can stop the script by typing kill . Next, remove the script from the server:

rm /home/accountname/public_html/forums/tmp/

Frozen messages with Exim

What are frozen messages?

Frozen messages are messages that Exim will no longer attempt to deliver. You can thaw the message with:
exim -Mt [ ... ]

To remove all frozen messages:

exiqgrep -z -i | xargs exim -Mrm
To show frozen messages:
exim -bp | grep frozen
Freeze all queued messages from local user:
exiqgrep -i -f luser@localhost | xargs exim -Mf

Finding Spam Scripts on cPanel

Finding Spam Scripts On cPanel

If a server is sending lots of spam, and no one knows why, there is a chance that the cause is a malicious script somewhere on the server. It can be a little tricky to locate such a script, but here's a trick that should help.

Log into WHM with root and the root password of the server.
Click on "Exim Configuration Editor" under "Service Configuration".
Click on the box at the top that says "Switch to Advanced Mode (Edit Raw Configuration File)".
Paste this into the top box:

log_selector = +address_rewrite +all_parents +arguments +connection_reject +delay_delivery +delivery_size +dnslist_defer +incoming_interface +incoming_port +lost_incoming_connection +queue_run +received_sender +received_recipients +retry_defer +sender_on_delivery +size_reject +skip_delivery +smtp_confirmation +smtp_connection +smtp_protocol_error +smtp_syntax_error +subject +tls_cipher +tls_peerdn

That line is pretty long, be sure to copy all of it and realize it may span beyond your browser's window. Most web browsers should allow you to "triple click" in the above field and copy all of the text to your clip board but if not simply place your mouse on the far left of "log_selector" and drag the mouse to your right until you have "+tls_peerdn" highlighted.

Once you've copied that into the Exim editor box, scroll all the way down and click the little "Save" button.

Now you're done in WHM. You can close out of the window, or leave it open if you plan on coming back to remove your addition to the Exim.conf. This change will slow Exim a little, so if you have a busy mailserver it's best to remove this modification when you're done.

Login to the server via ssh.

Watch the outgoing message log to see what directory messages are being sent from. This command works wonders:

tail -f /var/log/exim_mainlog | grep cwd

Note: cwd stands for current working directory.
This is quite normal: cwd=/var/spool/exim
This warrants investigation, but might be legit: cwd=/tmp
This is generally bad: cwd=/home/h4x0r/public_html/forums/tmp

Tuesday, March 4, 2008

How can I check which ports are open when the firewall is activated?

Use "netsh routing ip show filter" and
"netsh routing ip show filter "
to see open ports.

Saturday, March 1, 2008

Getting Protected from DoS Attacks with Apache

Note
This tutorial uses Apache 1.3. You can perform the steps for this on Apache 2.0, however you will need to follow the documentation for the applicable software discussed here.


Warning
This tutorial is not a substitute for a good firewall configuration, it is only an addition. Do not rely soley upon the information found here to prevent DoS attacks!


Firewall settings are great for preventing Denial of Service (DoS) attacks, however it may not always be your only solution. The day has finally arrived when I found this excellent module called mod_dosevasive (DoS Evasive) which keeps track of how many requests each client makes to your server within intervals. If a client is being forceful with your server and making too many requests, then it is more than likely not just a web browser but some automated process unleashed on your site to try and take it down.

This handy Apache module we have found takes care of these issues. Let's get started by setting it up.


Preparing for mod_dosevasive


The first step to prepare to install this module is determine your server configuration. Did you install Apache with Dynamic Shared Object support (DSO)? or is your server configured without it? If have DSO enabled, you can simply run the apxs command to import the module on the fly, otherwise you may have to recompile Apache to make this work.



Download and Modify mod_dosevasive


First, you need to download mod_dosevasive to your server. Do so by going to the official site:
Nuclear Elephant: DosEvasive

Download the source code to a directory of your choice and then unpack it:


cd /usr/local/src
wget 'http://www.nuclearelephant.com/projects/dosevasive/mod_dosevasive.1.8.tar.gz'
tar zxpf mod_dosevasive.1.8.tar.gz


Our next task is to determine if we want to use mod_dosevasive's email features. Later in this tutorial, we are going create an addon with PHP which allows us to do the mailing with a more verbose log of what is going on, and do the iptables firewall DROP rules, but but just in case, you need to edit the mod_dosevasive.c file and point it to the correct mailing application such as Sendmail.

In our mod_dosevasive.c file, we have altered the define MAILER line to reflect the path to Sendmail on our system:



#define MAILER "/usr/sbin/sendmail -t %s"



Now that we have the file prepared, we are ready to figure out how to get it into Apache.


Apache with DSO Support

If you're using an RPM version of Apache 1.3, then chances are your have DSO support enabled and this installation will be a breeze. First, locate the path to your apxs executable in the Apache bin directory and then run this command:


/usr/local/apache/bin/apxs -iac /usr/local/src/mod_dosevasive/mod_dosevasive.c


Watch for the output and if everything looks ok, the module is installed and you are clear to restart Apache. We'll add the configuration directives later in this tutorial, so you may skip the next section for now.


Apache without DSO

If your Apache is configured without DSO support (ie: not using --with-dso), then you must recompile Apache. Here's how to configure Apache with mod_dosevasive:


cd /usr/local/src/apache_1.3.29
./configure \
--prefix=/www \
--add-module=/usr/local/src/mod_dosevasive/mod_dosevasive.c
make
make install


Note
You will definately want to include any other Apache modules in the configure line that you will need. Don't forget to add PHP!


Now you are all set with Apache non-DSO and mod_dosevasive. Let's move on to the configuration files.

Basic Configuration mod_dosevasive

Our next task is to configure mod_dosevasive for your server. There's a good handful of directives you may use, but we'll cover the ones we need for this tutorial. You really should read the documentation on this module to understand the other directives that are available for you.


Apache 1.3 with DSO Support:

You need to add the module to the httpd.conf. Find the AddModule section and enter this line to the bottom of it:


AddModule mod_dosevasive.c



All Apache 1.3 Configurations:

At the end of your httpd.conf file, add the following lines:


Normal Traffic Server:




DOSHashTableSize 3097
DOSPageCount 2
DOSSiteCount 50
DOSPageInterval 1
DOSSiteInterval 1
DOSBlockingPeriod 10




High Traffic Server:




DOSHashTableSize 3097
DOSPageCount 4
DOSSiteCount 100
DOSPageInterval 1
DOSSiteInterval 1
DOSBlockingPeriod 10



Let's break down only these configuration directives:

DOSHashTableSize is the size of the hash table that is created for the IP addresses monitored.
DOSPageCount is the number of pages allowed to be loaded for the DOSPageInterval setting. In our case, 2 pages per 1 second before the IP gets flagged.
DOSSiteCount is the number of objects (ie: images, style sheets, javascripts, SSI, etc) allowed to be accessed in the DOSSiteInterval second. In our case, 50 objects per 1 second.
DOSPageInterval is the number of seconds the intervals are set for DOSPageCount
DOSSiteInterval is the number of seconds the intervals are set for DOSSiteCount
DOSBlockingPeriod is the number of seconds the IP address will recieve the Error 403 (Forbidden) page when they have been flagged.


The next task is to tweak Apache while we are here. Change the following directives to these values:

The MaxRequestsPerChild directive which is normally set to zero (0) for unlimited needs to be set for proper cleanup:


MaxRequestsPerChild 10000


Note
Read the documentation (README) again for this module to ensure you have any additional tweaks covered.


Moving along, we can now build our custom PHP script that will do some magic for us. Let's discuss this on the next page.

mod_dosevasive does a wonderful job as it is. However, we believe in the Three Strike rule instead of just dropping someone from your server alltogether after one flagging by mod_dosevasive. Therefore, we have created a method by using PHP and MySQL to handle this.

From this point forward we assume that you have PHP abilities on the command line. You can check to be sure by running:php -v and if the command is found, PHP will print it's version on the screen. You are clear to proceed!

Our setup will do a couple of things here. First, if mod_dosevasive finds an offending user, it has the ability to call an external command, in this case, our PHP script. We can pass the offending IP address to the PHP script and process it into our database. This provides ammunition for us if we need to because we can also keep track of how many times an IP address has been flagged and we can also log what pages it was flagged for accessing by parsing our Apache log file for the relative information. Then, we can have PHP execute the iptables command when we find this user has been flagged three times. Let's move on and find out how.


Setting up the MySQL Database


Note
Before you begin, you will need to have access to MySQL and know your MySQL root password!


First, login to MySQL and create a database, create a user and grant privelages to that database for the new user:


mysql -u root -pyourpasword
CREATE DATABASE blacklist;
use blacklist;

CREATE TABLE blacklist (
id int(5) NOT NULL auto_increment,
ip varchar(40) NOT NULL default '',
date datetime NOT NULL default '0000-00-00 00:00:00',
PRIMARY KEY (`id`)
) TYPE=MyISAM;

CREATE TABLE blacklist_logs (
id varchar(150) NOT NULL default '',
log_entry text NOT NULL,
ip_address varchar(40) NOT NULL default '',
date varchar(25) NOT NULL default '',
uri varchar(200) NOT NULL default '',
response varchar(100) NOT NULL default '',
size varchar(100) NOT NULL default '',
UNIQUE KEY id (`id`)
) TYPE=MyISAM;

GRANT ALL PRIVILEGES ON blacklist.* TO 'blacklist'@'localhost' IDENTIFIED BY 'somepassword';
exit



Now our database should be all set! We are ready to setup the PHP script.


The PHP Script


Our PHP script will perform a couple of operations:

Receive the IP address from the command line
Log the IP address, date and time into the blacklist table
Retreive the information about this user from the log file and log normal files such as HTML, PHP and etc into our database.
Count how many times the user has been flagged, if the count is 3 or more, then run the IPTABLES rule for dropping all access for this IP address.
Sending an email notification to an administrator with a summary of the logfile access we have recorded.


Sound pretty cool? Let's move on! Create a PHP script somewhere on your hard drive. I prefer to locate it in /usr/local/etc and name it blacklist.php with the following contents:



$sql = mysql_connect('localhost', 'blacklist', 'somepassword');
mysql_select_db('blacklist', $sql);

$ip = $_SERVER['argv']['1'];

$sql = mysql_query("INSERT INTO blacklist (ip, date) VALUES ('$ip', now())");
$logs = `cat /path/to/logs/access.log | grep $ip`;

$arry = explode("n", $logs);


foreach($arry AS $line){
if(!eregi('gif', $line) && !eregi('jpg', $line) && !eregi('bmp', $line)
&& !eregi('png', $line) && !eregi('css', $line)
&& !eregi('js', $line)){

$larr = explode(" ", $line);
$ip_addr = $larr[0];
$date = str_replace("[", "", $larr[3]);
$date = mytime($date, 0);
$url = $larr[6];
$response = $larr[8];
$size = $larr[9];
$id = md5($ip.$date.$url);
mysql_query("INSERT INTO blacklist_logs (
id, log_entry, ip_address, date, uri, response, size)
VALUES ('$id', '$line', '$ip', '$date', '$url$

}
}

function mytime($str, $short = null){
$nstr = explode('/', $str);
$nstr2 = explode(":", $nstr['2']);
$m = $nstr['1']; $d = $nstr['0']; $y = $nstr2['0'];
$h = $nstr2['1']; $i = $nstr2['2']; $s = $nstr2['3'];
if($short){
return "$d-$m-$y";
} else {
return "$d-$m-$y | $h:$i:$s";
}
}


$ccount = mysql_result(mysql_query("SELECT COUNT(id) AS ccount FROM blacklist WHERE ip = '$ip'"),0);


if($ccount > 2){
$drop = "su - root -c '/sbin/iptables -I INPUT -s $ip -j DROP'";
// echo $drop;
`$drop`;

$subject = "$ip Banned from Server";
$to = "you@yourdomain.com";
$message = "The IP address: $ip has been banned from yoursite.comnn";
$message .= "Here is a summary of actions for this IP address:nn";

$sql = mysql_query("SELECT * FROM blacklist_logs WHERE ip_address = '$ip' ORDER BY id");
while($row = mysql_fetch_array($sql)){
$message .= $row['log_entry']."n";
echo $row['log_entry']."n";
}
$message .= "====================================================================";
$headers .= "From: Securityn";
$headers .= "Return-Path: n"; //Return bouned mails to.
$headers .= "X-Priority: 1n"; //Message priority is set to HIGH
$headers .= "X-MSMail-Priority: Highn"; //Message Priority for Exchange Servers
$headers .= "X-Mailer: PHPn"; //IP address of who sent the mail.
mail($to, $subject, $message, $headers);
echo "$ip dropped and email sent.n";
}
?>



Note
Do not be a bafoon. Read each line of the previous PHP script and make sure the relevant information is correct for your server, usernames, email address and etc.

Note
For the previous script to work properly, you must be using combined logging methods within Apache: CustomLog /path/to/access.log combined


Now save this file out and you are ready to add another directive for mod_evasive on your Apache httpd.conf file:


DOSSystemCommand "/path/to/php /usr/local/etc/blacklist.php %s"


Now, restart Apache and watch as the script kicks into action. If you use phpMyAdmin, you can browse the database and also tail your system log: tail -f /var/log/messages

When an IP address gets blocked, you SHOULD receive an email notifying you if you setup the PHP script properly.

Note
If you restart your firewall, or Flush iptables, all of the IP address will be flushed along with it. However, the next time they get flagged, they should automatically get dropped from your firewall.

How to find file with 750 file permission?

How to find file with 750 file permission

# find / -type f -perm 750

Extra headers you enabled to help track Spammers

Extra headers you enabled to help track Spammers

Lets start by knowing where Exim keeps it logs
– Linux
● /var/log/exim_mainlog
– FreeBSD
● /var/log/exim/mainlog

Check the headers
– Is your hostname in there?
– Is your IP in there?
– Use the extra headers you enabled to help track down the spammer for you!

In the Exim Configuration add
– log_selector = +arguments + subject

Lets start by knowing where Exim keeps it logs
– Linux
● /var/log/exim_mainlog
– FreeBSD
● /var/log/exim/mainlog

It makes looking through the logs easier
– Take a look at the following example. You can
see exactly where the email was sent from.
● 2006-05-08 17:24
cwd=/home/user/public_html/phpBB 5 args: /
usr/sbin/exim -Mc 19Z8vf-0023mp-E2

ddos command

ddos command :


netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
route add 12.219.178.143 reject
route add 202.173.147.138 reject

===============

netstat -a | grep SYN | wc -l
iptables -A INPUT -s 124.177.148.139 -j DROP
netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

===============

Kill nobody /user process

Kill nobody /user process :

Kill nobody process
----
ps aux | grep nobody | awk '{print $2}'| xargs kill -9

Kill user process
----
ps aux | grep apache | awk '{print $2}' | xargs kill -9