Ubuntu Tricks: Text File Manipulation

You've got a text file that is several million lines. Each line corresponds to a filename and each file needs to either be downloaded or uploaded somewhere.

Assuming you have a script that iterates and reads in serial from the input text file, what happens when the system kicks out an error? Start from the beginning? Thats what happened initially for me while I looked for solutions. Yeah, just letting it go this way exponentially meant waiting three(3) hours and more each time. Thats not very optimal.

Since at the point of error it was possible to see the lines of text corresponding to the filename already or currently being processed, these should be the resume point. Noted this down.

What do we need to do? Trim the file from the top using sed and pipe out to a secondary file.

How? 

sed knows about line numbers, I haven't dug enough to make it do a string search in parallel, yet. The best way so far that worked for me is to use grep. Such that: cat | grep -n

             Example: 

             $ cat files2009.txt | grep -n meatloaf.bin

                4236321:meatloaf.bin 

So, we need to cut until line 4,236,321 (or perhaps a few lines before that, your choice). In case you haven't been tracking your percentage done that also tells us that we've just got less than 800K files to go before we're done.

Time to use sed which understands regular expressions. For our puposes we'll be using: sed -e ',d'  

             Example: 

             $ sed -e '1,4236321d' files2009.txt > new2009.txt

The above means to start trimming at line "1" up to "4236321" from files2009.txt and pipe the rest of the original contents to new2009.txt

You can now imagine how to trim blocks of lines from the middle, just cat for the starting line of text and so on.

Easy enough, and there you have it.

Ubuntu Tricks: Lock Screen Button

Saw this button some time ago in one of our internal systems. At first I thought it was an application that needed to be installed but after some searching and hindsight it didn't make any sense to actually need one more running program just to call the lock screen feature.

Fortunately, Ubuntu has this to say:

You can add a Lock Screen button to a panel for easy access. Right-click a panel (for example, the panel at the top of the screen), select Add to Panel... and drag the Lock Screen item to the desired location.

Works on my Jaunty, too.

Ubuntu Tricks: Disk Space Usage and Mount

Disk Usage and Size

Its simple, and I got it even working via ssh onto one of my work servers to check how much space was being used by a mounted USB drive. Just type the shell command: df

Of course this is going to give you the default output in bytes. This is more human readable using the switch -h provides the output in megabytes, while uppercase as -H shows the output in gigabytes.

Example #1: $ df -h
Output:
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/foo--bar-root
3.2T 2.5T 604G 81% /
tmpfs 1012M 0 1012M 0% /lib/init/rw
varrun 1012M 408K 1011M 1% /var/run
varlock 1012M 0 1012M 0% /var/lock
udev 1012M 2.9M 1009M 1% /dev
tmpfs 1012M 0 1012M 0% /dev/shm
lrm 1012M 2.0M 1010M 1% /lib/modules/2.6.27-9-generic/volatile
/dev/sda1 236M 52M 172M 24% /boot
//192.168.0.1/usbdrive
2.7T 2.3T 430G 85% /home/foo/storage

Example #2: $df -H
Output:
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/foo--bar-root
3.5T 2.7T 648G 81% /
tmpfs 1.1G 0 1.1G 0% /lib/init/rw
varrun 1.1G 418k 1.1G 1% /var/run
varlock 1.1G 0 1.1G 0% /var/lock
udev 1.1G 3.0M 1.1G 1% /dev
tmpfs 1.1G 0 1.1G 0% /dev/shm
lrm 1.1G 2.1M 1.1G 1% /lib/modules/2.6.27-9-generic/volatile
/dev/sda1 247M 55M 181M 24% /boot
//192.168.0.1/usbdrive
3.0T 2.6T 461G 85% /home/foo/storage

Subdirectory Size

What if you just wanted to see the size of a particular directory and subdirectories underneath it in the tree? Simply use the the command: du

Naturally, similar to df its going to default to bytes so just use the -h switch and you'll be fine.

Mount Shares
There are also several ways to mount a network share. Whats important is that the share needs to be configured for the permissions you need whether read/write/both/etc. Make sure that you've installed smbfs as using smbclient is rather the old way to do this:

mount -t smbfs //ip_address/path1/path2/... -o username=yourusername,password=yourpassword

Example: mount -t smbfs //192.168.0.2/foo/bar -o username=foo,password=bar

If you're trying to mount an external USB drive, then there's no better way than as explained here.

Of course, what comes on should be easy to come off. Enter umount:

umount -lfr /mount_point/samba_share

Example: umount -lfr /dev/sda1

The above syntax switches lets you remove the drive or whatever share no matter if its "busy" or if a network share that's "requiring administrative privilege" aka offline.

Have fun!

Ubuntu Tricks: Empty Directories

Have you ever used a tool that would go through subdirectories and delete the empty ones? One Windows I was able to do this with Heatsoft Clone Cleaner Pro (trial). But really, there has to be an easier way to do this without adding more software glut.

For those who like to manage their files, I do as part of work, here's a tip. If you can mount/map your main subdirectory onto Linux, all you have to do is issue this shell command:

find /path/you/specify -type d -empty -delete

It doesn't get any simpler than that. I did find however that in some instances when mixing symbolic links that it won't take. So you can try to change the target to a deeper directory level and issue the command from there. You can also go to the directory itself whereupon the simple format is to do: find ./ -type d -empty -delete

Simple and done. Happy!