Info

For those of you who know and love both Perl and regular expressions, that one-liner probably isn't too difficult to parse, but for the rest of you, let's go step by step. Sometimes it's easier to go through a one-liner if you see it in a formatted way, so here's the Perl part of the one-liner translated as though it were in a regular file:

HACK AND /

if(m|(A\d+\.\d+\.\d+\.\d+).*?01/Nov/2008|){ $v{$1}++;

First, let's discuss the while loop. Basically, while(<>) iterates over every line of input it receives either through a pipe or as a file argument on the command line. Inside this loop, I set up a regular expression to match and pull out an IP address. The regular expression is probably worth looking at in more detail:

This section of the regular expression matches the beginning of the line (A), then any amount of numbers (\d+), and then a dot, another series of numbers, another dot, another series of numbers, another dot and finally a fourth series of numbers. This pattern will match, for instance, 123.123.12.34 at the beginning of a line. I surrounded this part of the regular expression in parentheses. Because this is the first set of parentheses, when Perl matches it, it puts the resultant match into the $1 variable so I can pull it out later.

Now, those of you who know regular expressions know that I cheated here. This regular expression is not very explicit at all. For one, it would match completely invalid IP addresses, such as 999.999.999.999. For another, it even would match any series of four numbers with dots in between, such as 12345.6.7.8910. I chose an overly generic regular expression on purpose to make a point. There are explicit regular expressions that match only valid IP addresses, but those expressions are very long, very complex and, in this case, completely unnecessary.

Because I'm dealing with Apache logs, I am pretty confident that the first set of numbers at the beginning of the file is an IP address and not something else, and second, the IP address that Apache logged should be reasonably valid. In taking the shortcut, I not only saved on typing, but the resulting regular expression also is easier to read and understand even if you aren't a regex wizard.

After I match the IP, I want to match only log entries from November 01, 2008:

This section performs matches on any number of characters (.*), and with the question mark at the end, it matches only as much as it needs to and no more. Then, it matches the datestamp for November 01, 2008. If I wanted a tally of every day in the log file, I could omit this entire section of the regular expression. Alternatively, if I wanted to match on some other keyword (for instance, when the user performed a GET on a particular file), I could replace or augment the above section with that keyword.

Once I have matched the IP address in a line and have assigned it to $1, I then use it as a key in a hash I call %v here and increment it ($h{$1}++). The power of a hash is that it forces each key to be unique. That means each time I come across a new IP, it will get its own key in the hash and have its value incremented. So, if it's the first time I see the IP, its value will be one. The second time I see the IP, it will increment it to two and so on.

Once I'm done iterating through each line in the file, I then drop to a foreach loop:

Basically, all this does is increment through every key in the hash and output its value (the number of times I matched that IP in the file) and the IP itself. Note that I didn't sort the values here. I very well could have—Perl has powerful methods to sort output—but to make the code simpler and more flexible, I opted to pipe the output to the command-line sort command. That way, even if you don't know Perl too well but know the command line, you could tweak arguments in sort to reverse the output or even pipe it further to tail, so you could see only the top ten IPs.

If I want to know only the overall number of unique visitors, as each line represents a unique visitor, I just need to count the overall number of lines. To do this, I simply need to pipe the output to wc -l.

And, there you have it, a quick-and-dirty one-liner to chop through your logs and tally results. The beauty of using Perl hashes for this is that you can tweak the regular expression to match all sorts of values in the file—not just IP addresses— and tally all sorts of useful information. I've used modified versions of the script to count how many times a particular file was downloaded by unique IPs, and I've even used it to perform statistics on mailq output.■

Kyle Rankin is a Senior Systems Administrator in the San Francisco Bay Area and the author of a number of books, including KnoppixHacks and Ubuntu Hacks for O'Reilly Media. He is currently the president of the North Bay Linux Users' Group.

Wolfram Research's Mathematica

After rocketing its Mathematica application from version 6 to 7 in just 18 months, Wolfram Research's developers may need testing for blood-caffeine content. Mathematica is a powerful general computation environment for calculations, large-scale computations, complex programming, and visualizing and modeling data. After dubbing Mathematica 6 the "most important advance in its 20-year history", Wolfram says that version 7 is a major release that adds 500 new functions and 12 new application areas. Added functionality includes image processing, built-in parallel HPC, new on-demand curated data and other new computational innovations. The firm further claims that this release has made parallel computing mainstream. Mathematica 7 has 32- and 64-bit editions for Linux x86, Solaris UltraSPARC/x86, Windows and Mac OS X. www.wolfram.com

Neuros Technology's Neuros LINK and Neuros.TV Service

The Neuros folks definitely think like we do. Their new Neuros LINK is a hackable, nonproprietary set-top box that connects the television to the Internet via existing open Internet standards. Neuros positions the device "squarely between the dedicated, proprietary electronics devices and the powerful but clunky and expensive personal computer". One can play downloaded content in virtually any format from any source. Neuros also claims to have created a navigation structure that makes the LINK experience "feel like TV browsing rather than Web browsing". Currently, the LINK is a Gamma Product—that is, the post-beta, white box preproduc-tion stage especially geared for hackers and hard-core early adopters. Meanwhile, the accompanying Neuros.TV is a free service that enables Neuros LINK users to find, organize and share Web-based video content. open.neurostechnology.com

Same dog, new tricks, different owner. In other words, the popular Yellow Dog Linux (YDL) for the Cell Processor has been upgraded to version 6.1 and is now under the tutelage of the company Fixstars. The Tokyo-based Fixstars recently acquired Terra Soft Solutions, the company long associated with YDL. Fixstars states that YDL 6.1 is unique in that it drives both the desktop and development environments forward simultaneously. For end users, YDL v6.1 offers an improved graphical wireless configuration tool and the ability to use PS3 video RAM for temporary storage or swap. For developers, it offers advancements such as the new Cell Superscalar and support for the Cell's programming model. Supported platforms include Apple G4/G5, Sony Yellow Dog Linux™

PS3, YDL PowerStation and IBM Power Systems.

us.fixstars.com

NICTA and Infocomm Research's Maritime Wireless Mesh Network

Though still in the prototype stage, a wireless mesh network for ships has been announced by two research institutions: Australia's NICTA and Singapore's Institute for Infocomm Research. Under normal conditions, the maritime system provides data and voice communications between port authorities, container terminals and ships via shore-based WiMax inter-ship connectivity. In poor weather conditions, the system utilizes a backup satellite system. Because, says NICTA, standard VoIP and other data-transport techniques don't work well with satellite systems, the system utilizes onboard mesh nodes and NICTA's mobile routers to handle the satellite connectivity. The shore-based system aims to deliver a 6Mbps long-range (20km) ship-to-ship and ship-to-shore mesh communication system capable of ad hoc multi-hop communication with other vessels and shore command stations. www.i2r.a-star.edu.sg,www.nicta.com.au

fo&urOiS,

Fixstars' Yellow Dog Linux

Ingres Corp.'s Ingres Database

If we open-source enthusiasts have anything to brag about, it is a great selection of robust databases. A fine case in point is Ingres Corporation's Ingres Database, which was just upgraded to version 9.2. The company bills Ingres as "flexible, simple, secure, reliable and scalable [and able to] cope with even the most complex, multilanguage requirements including business intelligence, content management, data warehousing, enterprise resource planning (ERP) and logistics management". Core advancements in version 9.2 relate to improved application development, enhanced availability and supportability, as well as the simplification and automation of many tasks traditionally associated with maintaining a business-class database. www.ingres.com

Wiley Encyclopedia of Computer Science and Engineering

While the Web provides information in bulk at your fingertips, there remains no substitute for concise, authoritative reference works that are more than a stranger's brain dump. Such is the role of Benjamin W. Wah's new Wiley Encyclopedia of Science and Engineering, a five-volume, 3,300-page set with more than 450 A-to-Z articles on the latest advances and findings in computer science and engineering. Some broad topics include standards, electronic commerce, financial engineering and computer education. Each article is written by experts in their particular specialty and is peer-reviewed by two others to ensure reliability. www.wiley.com

Warren D. Sande and Carter Sande's Hello World! (Manning)

The father-son team of Warren D. Sande and Carter Sande think that anyone can program a computer, even a 12-year-old. The duo's new book Hello World! Computer Programming for Kids and Other Beginners from Manning is a "gentle but thorough introduction to the world of computer programming". Written in a manner free of "geek speak", Hello World! contains lots of pictures, cartoons and fun examples to hold the reader's interest. The free Python is the programming language utilized in the book. Programming concepts that are covered include memory, looping, decisions, input and output, data structures, graphics and others, which are then applied to interesting topics like computer graphics, game programming and simulations. The publisher says that Hello World! can be used in either a home or classroom setting. www.manning.com

Computer Pn ^rümmii^ Jor Kirfs and Other Re^frnners

Computer Pn ^rümmii^ Jor Kirfs and Other Re^frnners m wnrreii Orter Sflïid® , l\

AMAX's ServMax Personal Supercomputer

In support of the needs of scientific computing, AMAX has released its new ServMax Personal Supercomputer (PSC) workstation, which it dubs "a cluster in a box". The ServMax PSC supports up to 720 processing cores and 3 Teraflops in a single workstation. AMAX asserts that the product delivers "up to 15x cost savings and 15x lower power consumption than traditional 1U rack-optimized servers". Targeted applications include life sciences, geosciences, engineering and sciences, molecular biology, medical diagnostics, EDA, government/defense, visualization and financial modeling. Other features include parallel architecture and NVIDIA CUDA technology. www.amax.com

Please send information about releases of Linux-related products to [email protected] or New Products c/o Linux Journal, 1752 NW Market Street, #200, Seattle, WA 98107. Submissions are edited for length and content.

Fresh from the Labs

Eagle Mode—File Manager with a Difference

(eaglemode.sourceforge.net)

For those sick of file manager after file manager that essentially do the same thing, with only a slightly different interface, let's just say that Eagle Mode (EM) takes a more ambitious approach. In the words of EM's documentation:

Eagle Mode is an advanced solution for a futuristic style of man-machine communication, in which the user can visit almost everything simply by zooming in. It has a professional file manager, file viewers and players for most of the common file types, a chess game, a 3-D mines game, a multi-function clock and some fractal fun, all integrated in a virtual cosmos. By featuring a separate pop-up-zoomed control view, help texts in the things they are describing, editable bookmarks, multiple input methods, fast anti-aliased graphics, a virtually unlimited depth of panel tree, and by its portable C++ API, Eagle Mode aims to be a cutting edge of zoomable user interfaces.

Eagle Mode's file manager lives within a virtual cosmos where other programs and trinkets float around with it.

For those chasing a lightweight file manager, you're looking at the wrong project. I almost choked when I read the system requirements: CPU 3.4GHz...per core! Plus, 1GB of RAM and a gig of hard drive space for temporary files at runtime—this is an ambitious project that shoots at the opposite end of the scale! Nevertheless, don't despair if you don't have all the hardware requirements; I still found the project usable on my apparently measly 512MB of RAM and 2.14GHz per core.

Installation Thankfully, the software requirements aren't as Draconian as the hardware requirements. The only major dependencies are Perl, GCC, libx11-dev and the libxine-dev library for playing multimedia files. There is a list of other smaller non-essential dependencies that is long enough to make you scroll down the page (such as tar, xterm, JPEG libraries and so on), but they're really the kind of things you would expect for a file manager (and unless you have a system that's more sparse than a nightclub in Salt Lake City, you probably have them all installed anyway).

So, head to the Web site, grab the latest tarball, extract it and open a terminal in the new folder's main directory. From here, I really recommend reading the documentation included in the doc directory, which, unlike a lot of documentation, actually is well set out and easy to navigate. But, for the impatient, enter the following commands:

$ perl make.pl build

And, as root or sudo:

# perl make.pl install

I got an error in the middle of compiling EM, which said:

Building emAv failed, but that project is not so essential. So if you don't know how to solve the problem, you could continue the overall building now, and live without the features the project provides. Continue? [y(es)/n(o)/a(lways)]: y

After doing some Googling, I still couldn't find out what emAv was, but it seems to be non-essential, and the installation continued on unfazed. EM seems to run fine without it. Once the compilation has finished, change into the installation directory, which by default will be:

$ cd /usr/local/eaglemode/

And, enter this command to run it: $ ./eaglemode.sh

Usage Once inside EM's main screen, you'll be presented with a scene that is deceptively conventional—a file manager in your home directory. Double-click on a file, and it opens it. But, double-click on a folder, and it opens a terminal within that directory—that's weird. There's a bunch of extra info in the bottom right of the folder too, what's that about? As soon as I roll the mouse wheel upward, the whole scene suddenly zooms in, and I can see the contents of those files in detail. That's kind of cool, but it still isn't what I'd call groundbreaking. Then, I give the mouse wheel a few quick rolls backward, and the whole idea of Eagle Mode unfolds in an instant. When you zoom out, you find that everything is placed within a 3-D

Scroll backward all the way, and it is soon revealed that your virtual cosmos is living within the eye of this proud eagle!
I'm not sure if it will come out in print, but those who look very carefully will be able to see this as a white speck in its eye. Cosmic!

Well, worry no more radio DJs; this project may be just for you.

virtual cosmos—a star field consisting of files, applications and fields of stars. Keep zooming out, and eventually it is revealed that you are looking at a cosmos that is living within the eye of an eagle—just the kind of artistic silliness of which I approve! Zooming out to the end of the universe displays this proud eagle in its entirety.

Zoom back in to the eye of the eagle and you return to the virtual cosmos where you can zoom in to programs and folders in great detail. Clicking on an area of the universe changes the "aim" of the zoom, and if you hold the scroll wheel in and move the mouse in any direction, it will scroll around the screen that way. Included in these applications and objects are things like system folders, a clock, documentation and games, and I'm sure there are some hidden objects in that star field somewhere. For those looking for a distraction, a version of Chess has been included (with a rather aggressive AI, it must be said) and a game simply called Mines that is a mind-bending 3-D take on the classic Mine Sweeper.

Waste time in Eagle Mode playing a rather lovely 3-D Chess game against a rather nasty AI.

Overall, if you're sick of navigating your PC with something that feels like it was designed by a bank manager (or if you simply want to put all those GHz your new PC came with to use), Eagle Mode just might be for you (not recommended for boring people though).

Mp3Wrap— MP3 Merger

(mp3wrap.sourceforge.net)

With podcasting becoming ever more popular, people are dealing with large groups of MP3s that have to be squished into one big file and sent out to the general public. This in itself isn't very hard, but these files generally are all re-encoded and placed in something like a run-of-the-mill 128kbps MP3. When something that already has been under lossy file compression, like an MP3, gets encoded a second time, it loses a great deal of audio quality, and the resulting sound is more like a warbly old vinyl record being pumped through a Commodore 64. Well, worry no more radio DJs; this project may be just for you. According to the Mp3Wrap Web site:

Mp3Wrap is a free, independent alternative to AlbumWrap. It's a command-line utility that wraps quickly two or more MP3 files in one single large playable MP3, without losing filenames and ID3 information (and without need of decoding/encoding). It also provides the possibility of including other non-MP3 files, such as playlists, info files, cover images, inside the MP3. This means you obtain a large MP3 that you can split at any moment just using mp3splt, and in a few seconds, you have all the original files again! It's useful because files created with Mp3Wrap are easy to download. In fact, you don't need to know each song name to download, and it's easy to play. Even if you don't have mp3splt to split the file, you can listen to it anyway.

Installation Installing Mp3Wrap is a doddle, with a choice of a source

The mix-tape is back! Mp3Wrap lets you compile multiple MP3s into one large file without losing any sound quality.

tarball or .rpm and .deb packages. Plus, compiling the source is easy and painless. Click the DOWNLOAD link at the bottom of the home page for a list of all the package options. If you're going with the source version, grab the tarball, save it locally and extract it. Open a terminal in the new folder, and do your run-of-the-mill:

And, as root or sudo: # make install

Usage Mp3Wrap currently is a command-line-driven affair, but don't let that put you off, as it's quite simple. The syntax is as follows:

mp3wrap finaloutcomefile filetoadd1 filetoadd2

It looked like this after I had given it some files:

$ mp3wrap compilation1.mp3 *02-Origa_Ft_Shanti_Snyder-Rise-2004.mp3 11\ *Inner\ Universe.mp3

The terminal output does give some useful information as to what's happening. As a side note, remember that for some reason the program will insert _MP3WRAP just before the .mp3 extension as a sort of identifying mark, so if you're doing something like writing a shell script and having trouble finding your compilation MP3 file, that will be why.

A bunch of useful switches are included, the best of which is -a, allowing you to add more MP3 files to an existing "wrapped" MP3. Another useful switch is -l, which when passed a wrapped MP3 will list whatever files are inside. Check the man page for more details.

A drawback of the command-line nature of Mp3Wrap is that it may become very tiring and strenuous when dealing with a long list of MP3 files (which probably will lead to some mistakes with long playlists). Also, although Mp3Wrap's files are usable on just about anything that will play MP3s, they do have trouble seeking in some older players, such as XMMS and the like. This project is just begging for a GUI front end (which its cousin application mp3splt already has), as a GUI on top would make things much easier for a radio DJ on Friday night and would avoid the likely mistakes that will come from compiling a playlist of songs via command-line switches. Teething problems aside though, this program is a very clever one that will give podcasters a distinct edge over their rivals with original rip quality in their songs, and it might find its way into the hearts of many MySpace emo types looking to make an awful "mix-tape" MP3 compilation for some budding emo on-line girlfriend. Radio DJs and sad teenagers rejoice! ■

John Knight is a 24-year-old, drumming- and climbing-obsessed maniac from the world's most isolated city—Perth, Western Australia. He can usually be found either buried in an Audacity screen or thrashing a kick-drum beyond recognition.

Brewing something fresh, innovative or mind-bending? Send e-mail to [email protected].

Project at a Glance

Web Auction (apps.weblite.ca/webauction)

Sick of selling on eBay and getting slugged by seller fees? Or, would you simply like more control by having your own Web auction? Well, Web Auction (imaginative title, I know) might be exactly what you need. Designed for organizations or individuals, Web Auction is simple and quick to use, and it differs from eBay in that only administrators can add products, giving you full control over your auction. You can host an auction by yourself, but the Web Auction folks currently are letting you host auctions on their servers for free, which is jolly nice of them!

Web Auction

Expert included

Elizabeth, the product management expert for our rackmount server products, wanted to be here for this picture, but she is really busy these days. She's getting ready for the upcoming release of the newest Intel® Xeon® processor technology. Because Silicon Mechanics offers such a comprehensive product line, that means she's readying over 20 different products. Did we mention that she's busy?

Elizabeth knows that features such as Intel QuickPath Architecture, and DDR3 memory support will help our customers get more done in less time using less power than ever before. Want to know how? We'd love to have you call and talk it over with one of the experts at Silicon Mechanics for more information.

When you partner with Silicon Mechanics, you get more than the newest Intel processor technology— you get an expert like Elizabeth.

Was this article helpful?

0 0

Post a comment