UNIX Socket FAQ

A forum for questions and answers about network programming on Linux and all other Unix-like systems

You are not logged in.

#1 2008-02-28 07:46 AM

raviinroorkee.7
Member
Registered: 2007-04-25
Posts: 18

Re: Delete Files older than x hours

Can anyone suggest me a way for deleting files older than x hours.
Any command or shell script ?

Offline

#2 2008-02-28 01:37 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,826
Website

Re: Delete Files older than x hours

Just use something like tmpwatch...  That's exactly what it's designed for...

If you can't get tmpwatch (or similar) for whatever system you're on, then you could
roll your own with "find -mtime/-mmin/etc.", but it can get tricky...

Offline

#3 2008-03-13 09:05 AM

mlampkin
Administrator
From: Sol 3
Registered: 2002-06-12
Posts: 911
Website

Re: Delete Files older than x hours

Bored... and expanding on Rob's answer...

find . --min +N -name "*" -print | xargs /bin/rm -f

Where N is the number of minutes ( age ) minimum since the last modification time that must be matched...

Of course you could also use -cmin etc. instead... and the '.' part could be changed to the desired target directory... same with the "*" which current matches all files simply based on time ( ignoring name )...

As already warned... this can be VERY dangerous... a typo could result in a lot of files being deleted by accident... so changing the rm command to e.g. a mv so you could manually verify before deletion might be safer...


Michael


"The only difference between me and a madman is that I'm not mad."

Salvador Dali (1904-1989)

Offline

#4 2008-05-08 08:31 PM

jfriesne
Administrator
From: California
Registered: 2005-07-06
Posts: 348
Website

Re: Delete Files older than x hours

What I really want is to be able to set a "cachefile bit" on my file.  That bit would inform the filesystem that the file can be deleted if necessary to free up space on the drive.

With that feature in place, you'd no longer have to decide (e.g.) how many megabytes to devote to your web browser's cached-files directory, because it would be perfectly feasible for that directory to take up all the space on your drive.  The user would never notice, because he'd never see any "out of disk space" errors... instead, when the filesystem went to write to the disk and found that the disk was "out of space", it would simply find the oldest file with the cachefile bit set, delete it, and then try again.

If everybody's going to go around buying 1TB hard drives, we might as well use all of that extra space for something :^)

-Jeremy

Offline

#5 2008-05-08 09:17 PM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,230

Re: Delete Files older than x hours

That won't work out that well, because file placement is important too. Leaving
around old files could force the filesystem to use less optimal space for new files,
fragmentating them worse than necessary. Most filesystems perform rather badly
when the filesystem is almost full, so working on an always full disk would degrade
performance quite a lot. But I guess the above is mostly true for smaller filesystems.

Offline

#6 2008-05-08 09:26 PM

mlampkin
Administrator
From: Sol 3
Registered: 2002-06-12
Posts: 911
Website

Re: Delete Files older than x hours

Maybe something along the lines of FAM... and use of the sticky bit to indicate files that should NOT be removed...

?!

Michael


"The only difference between me and a madman is that I'm not mad."

Salvador Dali (1904-1989)

Offline

#7 2008-05-08 10:38 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,826
Website

Re: Delete Files older than x hours

Plus, at a certain large enough size, things like local disk-based caches will become
inefficient, anyway...  Ie: it'll end up taking longer to seek through the whole disk cache
to find what you're looking for (if it ends up even being in there after all that!) than it
would to just hit the network and download it again...  Exactly what that size is will vary
greatly depending on particular disk and network access specs, of course...  But, you
certainly wouldn't want to just blindly always give your whole disk away for cache
usage...  (RAM, on the other hand, sure...  Until networks VASTLY increase in speed,
anyway...  But, disks are still relatively slow and can easily be beaten by the network
at a certain load point, I'm sure...  Unless maybe you're still on 1200 baud dial-up... ;-))

Offline

#8 2008-05-08 10:39 PM

jfriesne
Administrator
From: California
Registered: 2005-07-06
Posts: 348
Website

Re: Delete Files older than x hours

Hmm, I see the point... perhaps the solution is allow the filesystem to auto-delete any 'cache-bit' file that gets in the way of where it wants to write.  That would solve the fragmentation problem, at the expense of having the decisions about which files to auto-delete be rather arbitrary instead of always being the oldest one.

-Jeremy

i3839;24445 wrote:

That won't work out that well, because file placement is important too. Leaving around old files could force the filesystem to use less optimal space for new files, fragmentating them worse than necessary. Most filesystems perform rather badly when the filesystem is almost full, so working on an always full disk would degrade performance quite a lot. But I guess the above is mostly true for smaller filesystems.

Offline

#9 2008-05-08 11:34 PM

mlampkin
Administrator
From: Sol 3
Registered: 2002-06-12
Posts: 911
Website

Re: Delete Files older than x hours

If there are lookup concerns...

The top N cache clusters e.g. keyed by site / domain could be kept in something like a skip list structure which would give very good lookup speeds for them...

The a secondary structure for them ( all entries ) could be kept in a hash or tree layout... more costly for time - especially if not perfectly optimized... but should still give a decent response time... it could also be dual keyed e.g. one set on site... the second on file age... the latter keying only existing if the site / files are allowed to be modified...

?

Of course its much easier writing this than it would be implementing it... lol

Michael


"The only difference between me and a madman is that I'm not mad."

Salvador Dali (1904-1989)

Offline

#10 2008-05-09 01:17 PM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,230

Re: Delete Files older than x hours

Jeremy, adding that functionality in the filesystem is only burdensome,
because it complicates them without adding any real advantage to do
it there instead of with a normal daemon. All the information you need
can be gathered by a daemon who can make intelligent decisions.
As for the cache bit, that can probably be done with extended attributes
or perhaps even easier, just keeping a list of directories which contain
cached data, then you don't even need application support.

Offline

#11 2008-05-09 03:22 PM

jfriesne
Administrator
From: California
Registered: 2005-07-06
Posts: 348
Website

Re: Delete Files older than x hours

Hi i3839,

You might be right, if there was some way for the daemon to get a callback when the filesystem is about to report an out-of-space condition to a user process, so that it can quickly free up some space before the error is reported to the user process.  I don't think that is possible, however... the best you could do (AFAIK) would be to poll the filesystem every so often and if it seems like it is getting "close to full", pre-emptively delete some deletable files in the hopes of preventing it from ever reaching 100% fullness.

There are several problems with that approach, however.  First, polling means that the daemon is using up resources even when there is no work to be done.  Second, because the daemon's operation is asynchronous with the other processes on the system, there is still no guarantee that it will work:  if another task is writing data quickly, it could still run out of disk space before the daemon had time to wake up, notice the low-space condition, and free some space.  Third, you'd have to make a tradeoff decision about how much free space to maintain... too much, and you lose the benefit of the mechanism; too little, and you have a higher risk of having a user process experience out-of-space errors that it shouldn't have experienced.

An in-fileystem implementation would not have these problems, since it could do something like this:

int write(...)
{
   int ret = do_write(...);
   if ((ret < 0)&&(errno == ENOSPC))
   {
      DeleteSomeAutoDeleteableFiles();
      return do_write(...);
   }
   return ret;
}

-Jeremy

i3839;24455 wrote:

Jeremy, adding that functionality in the filesystem is only burdensome, because it complicates them without adding any real advantage to do it there instead of with a normal daemon. All the information you need can be gathered by a daemon who can make intelligent decisions.  As for the cache bit, that can probably be done with extended attributes or perhaps even easier, just keeping a list of directories which contain cached data, then you don't even need application support.

Offline

#12 2008-05-09 04:53 PM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,230

Re: Delete Files older than x hours

True, except that harddisk are so damn slow that in practice it's not a problem,
because you will be able to delete files much quicker than filling your disk.
Harddisk also don't fill up very suddenly, so you can see it comming for a while.
So you can check free disk space, see that it has 5 Gb free, and check again in
a few minutes. If it has 50 Gb free you can check in a few hours. The closer the disk
comes to being full, the more often you need to check. How much free space is left
over in the end determines how often you need to check. Of course whacky
applications can seek around in files like crazy, expanding it beyond the end of
the file and increasing the amount of reserved disk space for it, for filesystems
that do the accounting immediately, but it's doubtful you're better off with it
succeeding.

In the end it's a matter how much information needs to be exchanged. The fs
knows when the disk is full, but everything else is known by user space. So an
optimal implementation would probably be a daemon or program that is notified
or run by the kernel when the disk is full and space is needed. But I suspect that
even just that would be overkill.

Edit: Checking the diskspace usage is only one statfs() call, so it's very cheap anyway.

Offline

Board footer

Powered by FluxBB