Web Hosting Forum | Lunarpages


*
Welcome, Guest. Please login or register.
Did you miss your activation email?



Login with username, password and session length
February 09, 2012, 05:59:33 PM

Pages: [1] 2   Go Down
  Print  
Author Topic: Files & Folders Monitoring Tool  (Read 2821 times)
MajorDad
Newbie
*
Offline Offline

Posts: 2


« on: February 02, 2007, 05:00:05 AM »

I was wondering...

I keep watching my whole site's files and folders for signs of intrusions (new files or folders that aren't supposed to be there) and thought that there must be a monitoring tool to do that for me, and send an email alert or any other action.

I know it exists for Windows, I've searched and found several tools that do exactly that. Now, I was wondering if anybody knows of any tool/scripts made to run on the LunarPages servers and if it is possible to use them.

Thanks!
Guy
Logged
SteveW
Master Jedi
*****
Offline Offline

Posts: 1394


WWW
« Reply #1 on: February 02, 2007, 09:26:25 AM »

I don't know the answer, but that's a great idea.

If one doesn't already exist, I wonder if it's possible to accomplish a similar thing using already available tools: Create a cron job that lists all the files and folders in your site. I'm not much familiar with Linux/Unix, but I think the command is "ls" or some variant, maybe with switches? Then have the results emailed to you. Use Windows/DOS fc.exe to compare each day's listing against the previous.
Logged





Mt. Shasta
photo gallery.


Don't forget Lunarpages 24/7/365 support documentation:
Flash Tutorials, Knowledge Base FAQ Articles, cPanel Manual, Glossary/Dictionary, Support Tickets,
and
Forum Search.

tarheit
Galactic Royalty
*****
Offline Offline

Posts: 467


WWW
« Reply #2 on: February 02, 2007, 09:28:48 AM »

If you backup regularly, it might be easier on the server if you compare the backups for changed/modified files.    A quick google search turned up several programs that are suppose to compair zip/tar/gz files.

See http://www.freedownloadscenter.com/Utilities/File_Synchronizing_Utilities/ArcDiff.html for one.

-Tim
Logged

SteveW
Master Jedi
*****
Offline Offline

Posts: 1394


WWW
« Reply #3 on: February 02, 2007, 10:22:47 AM »

The specific use I was thinking of wouldn't really require daily runs. When someone posts here after they've been hacked, one of the standard bits of advice is to "go through your entire site looking for files you didn't put there". That is an incredible pain, even using FTP, and besides, there are numerous files that were not put there by the site owner but are nonetheless ok and necessary.

So I was thinking why not do a file listing at a point where you know you haven't been hacked. Then you have a clean listing for comparison after a hack.
Logged





Mt. Shasta
photo gallery.


Don't forget Lunarpages 24/7/365 support documentation:
Flash Tutorials, Knowledge Base FAQ Articles, cPanel Manual, Glossary/Dictionary, Support Tickets,
and
Forum Search.

leighsww
* The Tough Love Cuddly One *
Berserker Poster
*****
Offline Offline

Posts: 13870


WWW
« Reply #4 on: February 02, 2007, 11:36:59 AM »

I found this "Tripwire" tool that looks like it will do the trick --> Click here

Here's the open source link:

http://sourceforge.net/project/showfiles.php?group_id=3130

The only thing that we might need some help with from geeks who know this stuff is  Smile ...

There are two versions, which version would be the best one to download to be compatible with LP's servers?

The first one says:

Quote
The x86 bin package consists of binaries built on a gentoo 2005.1 system with --enable-static.

When I looked up "Gentoo" it had to do with Linux, but I'm thinking it must be just a version of Linux, and that the 2nd version is also a Linux version.

The 2nd one says "Architecture = Any" and the first one "Architecture = i386" so I guess the 2nd one would be compatible on any Linux system and the 1st one only on an "i386" Linux system. Is LP an "i386"?

Anyway, if somebody who knows this stuff can give feedback on which version we should download, that would be great! Thanks!  Luff Ya
« Last Edit: February 02, 2007, 11:53:08 AM by leighsww » Logged
GMTurner
Berserker Poster
*****
Offline Offline

Posts: 7539



WWW
« Reply #5 on: February 02, 2007, 12:56:45 PM »

building on SteveW's idea... I wonder if it would be possible to have the results of the directory listing (path, filename, and last modified date mainly) placed into a database. If it is a new file then it is added as a new record. If a file disappears it is marked as "deleted". Otherwise just the modification date is updated. Then after a hack you could sort the files based on the last modified date and see both what new files had been added as well as which files had been modified and when. For something like this, you'd probably need to be careful about what directory listings you were getting and how often since it could take some processing depending on the number of files and subdirectories... you could even use the opendir(), readdir(), etc. commands in PHP to handle this without having to use cron, passthru(), etc. to run something at the command-line level... now if I find the time I might play with this idea a bit...
Logged

The above post was made at a time when I gave a dang and doesn't necessarily reflect my current views or opinions.

For those no longer with us ... Grr..!!

Turner's Lounge
SteveW
Master Jedi
*****
Offline Offline

Posts: 1394


WWW
« Reply #6 on: February 03, 2007, 12:58:52 PM »

Leigh, I looked at the Tripwire info, and would hesitate to try to install it unless someone posts here to say they are already using it successfully and can offer help setting it up. It looks like overkill for what we'd need here. If there are any bugs at all (there are some), it could destabilize the shared server, not good. And it might not be allowable to install on a shared server, anyway. The database it builds couldn't be kept on a secure floppy disk as intended, and thus it could be hacked or erased by a hacker.

To your specific questions:

The one that says "Any" architecture is the source code; you'd have to compile it yourself. The i386 one is already compiled (it is executable) and is the one you'd download, and it refers to Intel 386 architecture (and probably compatibles such as AMD but maybe not). That should mean i386 or later, i.e. pretty much any modern computer. So basically all our servers are most likely i386, and the program does appear to be intended for Linux, so it "should" have the potential to work, but that wouldn't give me sufficient confidence to experiment with it on a server with 500 other websites on it.

Static probably means static linking, meaning everything the program needs is compiled into the program itself, and it doesn't rely on external libraries. That's a good thing.

I've done a preliminary look at "LS", and I think that might do the trick if it's allowed as a cron job. I think the command line will be something like the following. It should be all lowercase. I used upper to make the letters distinct:

LS -LTASD path

I'm not sure what path will be. The ideal  target would be your root directory, just above public_html, to get ALL the files in your account. But in the context of LS, I don't know how you specify that.

I also don't see yet how to make it recurse subdirectories so it produces a complete report of your entire account.


« Last Edit: February 03, 2007, 01:15:40 PM by SteveW » Logged





Mt. Shasta
photo gallery.


Don't forget Lunarpages 24/7/365 support documentation:
Flash Tutorials, Knowledge Base FAQ Articles, cPanel Manual, Glossary/Dictionary, Support Tickets,
and
Forum Search.

Lupine1647
Berserker Poster
*****
Offline Offline

Posts: 10868


« Reply #7 on: February 03, 2007, 01:09:31 PM »

You wouldn't be able to install Tripwire on our shared servers as that will require a compile and install on the server. The best thing to do on a shared server would be to do some sort of ls, have it record the files (or do an md5 hash of the list of files). If the list changes at all it won't match the md5 hash in the database or where every you stored it at and then you would know the file structure has changed.

You can create a PHP script that would generate a list of files and directories for each directory or a list of specified directories to get a hash from and to compare to at a later time (that way you know specifically where the files may have changed instead of having to go through each directory).

Let us know if you have any other questions.
Logged
SteveW
Master Jedi
*****
Offline Offline

Posts: 1394


WWW
« Reply #8 on: February 03, 2007, 01:14:02 PM »

Exploring this further is on the list of "things I probably can't get to today", so if anyone wishes to go on ahead, the To Do list is:

1) Find the LS command, if any, to recurse subdirectories. If none, the project's over.

2) Figure out the path to root. It might be something simple like "/". Or maybe "YourUserID/" or "/YourUserID/ ?

2a) If that isn't allowed, figure out the path to /public_html.

3) Figure out how to set up a cron job... with results saved to a file? Or with results emailed?

4) Then set up a cron job to test the command line.

5) See if it worked.

Thanks, RyanJ, sounds like this might be a productive route after all.

Edit: if anyone knows the answers to some of those questions, it would help, but as it is, there are too many "this is the first time I've done this" steps, and I can't spend all day today studying new things, which is how a project like this tends to progress...
« Last Edit: February 03, 2007, 01:22:33 PM by SteveW » Logged





Mt. Shasta
photo gallery.


Don't forget Lunarpages 24/7/365 support documentation:
Flash Tutorials, Knowledge Base FAQ Articles, cPanel Manual, Glossary/Dictionary, Support Tickets,
and
Forum Search.

MrPhil
Berserker Poster
*****
Offline Offline

Posts: 5083



« Reply #9 on: February 03, 2007, 03:17:19 PM »

Exploring this further is on the list of "things I probably can't get to today", so if anyone wishes to go on ahead, the To Do list is:

1) Find the LS command, if any, to recurse subdirectories. If none, the project's over.

I don't have Linux booted right now, but IIRC, the switch is -R to recursively list directories. Don't confuse it with -r, which reverses the listing order. There's no such word as "recurse" (where's the emoticon for "grind teeth"?).

Quote from: SteveW
2) Figure out the path to root. It might be something simple like "/". Or maybe "YourUserID/" or "/YourUserID/ ?

2a) If that isn't allowed, figure out the path to /public_html.

I think bash, Perl, and PHP scripts will all consider / to be the system root, so you probably want to start at /home/<your account>/.

Quote from: SteveW
3) Figure out how to set up a cron job... with results saved to a file? Or with results emailed?

4) Then set up a cron job to test the command line.

I think the cron command could be as simple as ls -alR. Let it email the whole output to you. On your PC, compare it against an older, "good" listing.

Quote from: SteveW
5) See if it worked.

Thanks, RyanJ, sounds like this might be a productive route after all.

Edit: if anyone knows the answers to some of those questions, it would help, but as it is, there are too many "this is the first time I've done this" steps, and I can't spend all day today studying new things, which is how a project like this tends to progress...

If you can pull the listing itself out of the email into a file on your PC, you could try to find a "smart-compare" utility like the Linux diff to highlight differences between the new and old files. Don't bother with the DOS comp command -- it's worse than useless. That reminds me, I need to take a stroll over to Sourceforge or FSF to see if they have anything like diff that works on Windoze.

More advanced work would be to do a checksum on each file, to see if someone has altered a file without changing its size or timestamp. LP might have the md5 (?) command installed. I don't know if automating any of this would be worth the effort, as files do change, appear, and disappear during normal operations. Maybe you could write something to just flag new files and changes to certain existing files (e.g., .html or .php, but not maiboxes). You could also have all this work on on the server side (do an ls and/or md5 on the files, compare it to the previous, and email back any suspicious changes).
« Last Edit: February 03, 2007, 03:25:01 PM by MrPhil » Logged

SteveW
Master Jedi
*****
Offline Offline

Posts: 1394


WWW
« Reply #10 on: February 03, 2007, 07:33:51 PM »

MrPhil, thanks for all that info.

There's no such word as "recurse" (where's the emoticon for "grind teeth"?).
Apologies to your teeth. But... but, I see it everywhere, and never thought to check on it. What if I curse a directory once, and then curse it again?

Quote
the DOS comp command
FC (aka filcom or filcomp?) is the one I was thinking of, probably the same. In the case of a hack, though, I'd be more likely to take the FTC approach (Fine-Toothed-Comb).
Logged





Mt. Shasta
photo gallery.


Don't forget Lunarpages 24/7/365 support documentation:
Flash Tutorials, Knowledge Base FAQ Articles, cPanel Manual, Glossary/Dictionary, Support Tickets,
and
Forum Search.

GMTurner
Berserker Poster
*****
Offline Offline

Posts: 7539



WWW
« Reply #11 on: February 06, 2007, 12:13:13 AM »

here's the code I've been working on for myself related to this... it is a PHP script that generates a listing of files, sorted by modification time. This means the more recent files will be towards the top of the list. I've set it up with some configurable options at the start for no real reason...

At a minimum you will need to set the value of $cPanel_UserName to your account. If you do not limit it to public_html, then you will get a lot of "unexpected" files (mailboxes, various cpanel files, log files, etc) which are modified each day. You can limit the number of days back it goes, though really it is basing it on 24 hour periods, so if you run it at noon set to go 1 day back, it only goes back to noon on the previous day not midnight. Still, this could be useful to find files added/changed on your site in the past xx days. The "formatted" output isn't exactly pretty, but it's functional. The comma delimited version is best viewed as source and creates a file you could import into Excel or another program if you wanted to.

also, I do not consider myself a programmer, so I'm sure the code could probably be improved by someone who knows what they are doing Smile Use at your own risk...

Code:
<?php
$cPanel_UserName 
"";
$start_in ""// leave blank to start in the root of your account, otherwise enter the directory name
$how_far_back 0// set to 0 to list all files, or enter number of days to go back
$timeformat "Y-m-d h:iA"// php date format, see http://us2.php.net/manual/en/function.date.php
$displaystyle 0// 0=table formatted, 1=comma delimited (HINT: Use view source if choose 1)

$fileinfo = array();
if ( 
$how_far_back !=) {
$how_far_back=time()-($how_far_back*24*60*60);
};

function 
LinkFiles($a$b) { // reads filenames (and paths) into array
  
global $fileinfo$how_far_back;
  
$dir opendir($a);
  while (
$file_name readdir($dir)) {
    if (
$file_name != "." && $file_name != ".." && $file_name!="www") { 
    
// www listing is removed because it is a sym_link to public_html and creates "duplicates"
    
 $test "$a$file_name";
      if ( !
is_file($test) ) {
$sub_dir "$a$file_name/";
LinkFiles("$sub_dir""$b/$file_name");
      } else {
      
$fn "$a$file_name";
      
$ft filemtime("$fn");
      
$fs filesize("$fn");
      
if ( $how_far_back<$ft ) {
       $fileinfo[] = array( array( FTime => "$ft"FName => "$fn"FSize => "$fs") );
     };
      };
    };
  };
  
closedir($dir);
};

if ((
trim($start_in)!="") && (substr("$start_in",-1)!="/")) { $start_in .= "/"; };
$main_dir "/home/$cPanel_UserName/$start_in"
LinkFiles($main_dir$start_in);

array_multisort($fileinfoSORT_DESC); // sort it by date with new files at top

if ($displaystyle==0) {
echo "<p><font size=+3 color=0000ff>File Listing for: $main_dir<br>";
echo "Files modified: ".date("$timeformat"$how_far_back)." to ".date("$timeformat"time())."</font></p>";
echo "<table border=0 cellspacing=0 cellpadding=2>";
foreach ($fileinfo as $file) {
echo "<tr>";
echo "<td style='border-bottom: 1px solid #000000'><pre>".date("$timeformat"$file[0]['FTime'])."    </pre></td>";
echo "<td align=right style='border-bottom: 1px solid #000000'><pre> ".$file[0]['FSize']."</pre></td>";
echo "<td style='border-bottom: 1px solid #000000'><pre>  ".$file[0]['FName']."</pre></td>";
echo "</tr>\n";
};
echo "</table>";
echo "Listing complete...";
} else {
echo "Timestamp, File Size, Path/File\n";
foreach ($fileinfo as $file) {
echo date("$timeformat"$file[0]['FTime']).", ";
echo $file[0]['FSize'].", ";
echo $file[0]['FName']."\n";
};
};
?>


Again, didn't say the code was pretty, but it gets the job done and might serve as a starting point for your own project... my eventual goal with this is to have the info inserted into a DB and the script can check against the DB to determine which files have been modified. Once I identify files/directories that change frequently (like the mail inbox) I will eliminate those from being checked. Then the script can be set to run via cron once a day or so and email a list of files that have changed since the prior day... of course, getting the script to that point is going to take me some time since this is really low on my list of to-dos at this point... just my thought on how to go about doing this without having to execute commands via cron or compare files to each other for changes.... in the short run though I can check files against a list I've already generated and look back for files modified in the past week or so if I ever need to try and track something down...

It is now late (my time) so ignore the parts of this post that make absolutely no sense Smile
Logged

The above post was made at a time when I gave a dang and doesn't necessarily reflect my current views or opinions.

For those no longer with us ... Grr..!!

Turner's Lounge
GMTurner
Berserker Poster
*****
Offline Offline

Posts: 7539



WWW
« Reply #12 on: February 06, 2007, 12:15:16 AM »

Oh, and the function is called LinkFiles because it's actually a modified version of a function I had in another script and I forgot to change the name to something more appropriate... the other script actually creates links to files in a sub-directory so faculty I support could easily add files to their website by just copying them into a folder... of course I don't think any of the faculty are still using that script... it's a few years old... but that's unrelated to this and I'm rambling because I'm half asleep...

Logged

The above post was made at a time when I gave a dang and doesn't necessarily reflect my current views or opinions.

For those no longer with us ... Grr..!!

Turner's Lounge
SteveW
Master Jedi
*****
Offline Offline

Posts: 1394


WWW
« Reply #13 on: February 06, 2007, 08:20:42 PM »

 Happy Happy Joy Joy
To set up a cron job, go to cpanel > Cron Jobs > Standard. Enter the email address to use and the command line to run. For the other fields, I made selections in all of them, corresponding to "right now", i.e. Tuesday, February 6, 7PM, with the Minutes field set safely to several minutes in the future. Then click "Save Crontab". After this one-time cron job has run, be sure to go back into Cron Jobs > Standard and delete it.

As an experiment, I set up a cron job with the command:
pwd
which shows the current directory. This showed that I was in my home (root) directory (/home/myusername), and that therefore I could run the ls command with no path specified, as the default one would be correct.

To get a complete listing of all the files in your site, you can use this command line with exactly this capitalization:
Code:
ls -1aFlqR

Some other options are listed at http://www.ss64.com/bash/ls.html and http://en.wikipedia.org/wiki/Ls.

The resulting email can be big. My listing was about 1MB with 13,000 files listed. The fine-toothed-comb comparison method is thus impractical. Haven't considered yet how to compare the two files. Probably with FC.EXE to start with, and MSAccess might provide some useful methods such as with a Find Unmatched query.

I opted to use the default sort order for the ls command, and NOT sort by last-modified timestamp, because if you're going to compare TWO of these listings, you want the files listed in the exact same order.

Now that I have the file listing I need, I don't plan to run this cron job at any specified interval such as to discover a hack. I figure the fact that a hack has taken place will be obvious enough that one doesn't need to hunt for it. This listing is only for identifying suspicious files to examine after you know a hack has occurred.

Here are some sample output lines:

Code:
drwx--x--x  497 root     root        12288 Feb  5 00:06 ../
drwxrwx---    3 UserID   mail         4096 Feb  3 11:20 mail/
drwxr-x---    3 UserID   UserID       4096 Jan  6  2006 public_ftp/
drwxr-x---   33 UserID   nobody       4096 Feb  5 20:51 public_html/

The first character is the file type, "-" for file, "d" for directory, "l" for link. The remaining 9 characters are the permissions levels.
I don't know what the 2nd column is.
The 3rd and 4th columns must have to do with who created the file or who owns it or who last modified it? The next column is the file size, then the month and day. The next column will be a misery for a database import: it's the last-modified timestamp unless it's old, in which case it's the year! The last column is the file name, followed by a "/" if it's a directory rather than a file.
« Last Edit: February 06, 2007, 08:41:57 PM by SteveW » Logged





Mt. Shasta
photo gallery.


Don't forget Lunarpages 24/7/365 support documentation:
Flash Tutorials, Knowledge Base FAQ Articles, cPanel Manual, Glossary/Dictionary, Support Tickets,
and
Forum Search.

ronmsf2
Master Jedi
*****
Offline Offline

Posts: 1174


« Reply #14 on: February 06, 2007, 09:14:22 PM »

If you can pull the listing itself out of the email into a file on your PC, you could try to find a "smart-compare" utility like the Linux diff to highlight differences between the new and old files. Don't bother with the DOS comp command -- it's worse than useless. That reminds me, I need to take a stroll over to Sourceforge or FSF to see if they have anything like diff that works on Windoze.

Winmerge is great for comparing text files.  Not sure if it will work for what you're interested in, but is certainly worth checking out.

http://winmerge.org/

Logged

"Be glad of life because it gives you the chance to love and to work and to play and to look up at the stars" -Henry Van Dyke

"Education is the kindling of a flame, not the filling of a vessel." -Socrates
Pages: [1] 2   Go Up
  Print  
 
Jump to: