Klixxx Magazine Archive - Using LWP Modules with Perl

html hit counter

Klixxx Adult Webmaster Network

Klixxx Magazine Archive - Tech "Know-How"

Using LWP Modules with Perl *
by Tim Schaefer

The Problem
Have you ever wanted to be able to pull something off of one Web site and use it in a program for your Web site? The only way it seems possible is to have some kind of macro-type program doing repetitive tasks through a web browser, but maybe, just maybe if you had a program that would act like a web browser you would be able to do much more than simply play keystrokes. Macro programs have their uses but can often lead to a lot of frustration and extra work.

The Solution
The magic of the Perl programming language once again can be used to help you, allowing you to create web-browser behavior right inside your Perl programs. In this article we'll talk about the magic of using the LWP module, a series of tools that turn your Perl program into a web browser. LWP creates a web browser object for you, goes to the Web site of your choice, and pulls back the HTML just as if you were sitting there with a Netscape or Internet Explorer or other web browser. Don't worry, you don't have to be a rocket scientist to use LWP, what you really need is a good example and you'll be using LWP for all kinds of projects.

An Example
As always a good example is necessary. Recently I needed to create a United States zip code database table but could not find this data anywhere in a form that I could load into a database. Solution? Create my own with LWP and Perl. I found the USPS Web site very useful, but could not see myself sitting and inputting 100000 zip codes into this form:

Instead, I thought of using a program to gather this information automatically with Perl and LWP.

The Perl program I created runs a loop of approximately 100000 hits to the USPS Web site, looking for city and state when I send it a zipcode. The USPS does not have 100000 zipcodes, more like half that, give or take a few, but since I don't know which ones are valid and which ones are not, I'll have to hit the site 100000 times. While this may seem silly, and give the appearance of being a bad-boy hitting their site so many times, the USPS must know that millions of hits come in to their web site every day from people using their web browsers in a normal fashion. Since LWP behaves as a web browser we are only doing the same thing as a person sitting at a web browser, only the computer is doing the repetitive task of looking at the USPS Web site 100000 times.

The above diagram shows basically what is happening when you use an LWP-enabled Perl program. In the above diagram Server A hits Server B, but you can actually get creative and have both servers talking to each other posting back and forth using LWP requests to retrieve data from each other across the web. In the case of my zip code program, Server A is my little Linux box at home, and Server B is the USPS web site.

The Perl Program
Onward to the code. If you are not an experienced Perl programmer, fear not, this program is written in a simple way to show you what to do. Read the comments throughout the program and follow it to the end.

You will need to make sure you have the LWP:: modules installed on your system in order to use them. These can be found at www.cpan.org.

#!/usr/bin/perl
################################################################################
#
#     Program:  get_usps_zip_codes.pl
#      Author:  Tim Schaefer, Data Design Technologies, Inc.
#       Email:  tim@datad.com
#     Created:  2001-01-01
# Description:  Retrieves city and state names from the USPS Web site
#               by sending a 5-digit zip_code using Perl and LWP:: modules.
#
#       Usage:  get_usps_zip_codes.pl
#      Output:  A file called USPS_ZIP_CODES.TXT, which contains city, state, 
#               and 5-digit zip-code in a pipe-delimited format suitable for 
#               loading into a database.
#
################################################################################
# $Revision: $
#   $Author: $
#    $State: $
#     $Date: $
#   $Locker: $
#      $Log: $
################################################################################

use LWP::UserAgent;
use LWP::Simple;

$|=1;

my $zip_code;

get_all_towns();

sub get_all_towns {

#
# Runs a loop through all possible USPS zip_codes
# and calls the get_one_town function to produce a 
# city and state for that zip_code
#

#
# Change the name of the output file if you don't
# like this one.
#

my $zip_code_file = "USPS_ZIP_CODES.TXT" ;
	open ( ZIP, ">$zip_code_file" );

#
# Count is set to 500 but could be set to anything
# if you only wanted to start at a certain number
#

$count =  500 ;

while ( $count < 100000 )
		{

#
		# Format the zipcode with leading zeros
		#

$zip_code = sprintf ( "%5.5d", $count ) ;

#
		# For learning purposes watch and see
		# by un-commenting this print statement
		#

print "$zip_code\n" ;

$locality = get_city_and_state( $zip_code );

@local_arr = split /:/, $locality ;  
		$g_city  = $local_arr[0] ;
		$g_state = $local_arr[1] ;
		if ( $g_city && $g_state )
			{
			print ZIP "$g_city\|$g_state\|$zip_code\|\n" ;
			}

++$count ;

}

close ( ZIP ) ;

}
1;

sub get_city_and_state {

#
# If there is a zip_code sent to this function do the following
#
	my $zc = $_[0];           # the zip_code

my $break_count  = 0 ;
	my $capture_flag = 0 ;
	my $city_state_string = "" ;
	my $i            = 0 ;
	my $cs_count     = 0 ;
	my $token_string = "" ;
	my $token_add = 0 ;

if ( $zc )                              
		{

#
		# Call the LWP module to pull back HTML to 
		# parse, hunting for the city and state.  
		#

my $info = post_to_server( $zc );   # LWP module

#
		# If you want to see the HTML, print it:
		#
		# print "$info\n" ;
		#

#
		# separate each HTML code by putting a space
		# on the < and > so that we can examine each
		# item in the $info.
		#

$info =~ s/\</\ </g ;
		$info =~ s/\>/> /g ;

#
		# put all the items and words into an array that we
		# can examine using the map command.
		#

my @info_arr = map { split ' ' } $info ;

#
		# walk the array looking for the city and state
		#

foreach $token ( @info_arr ) 
			{

#
			# UPPER-case the token 
			#

$token = uc($token);

#
			# use this print statement to demonstrate and display
			#
			# print "token: $token\n" ;
			#

# 
			#  As we walk through the text we need to mark
			#  a starting point that is always predictable near
			#  where the city and state will be found.   The
			#  word "TYPE" is predictably found right before
			#  the city and state every time we get a valid 
			#  zip_code sent to the usps.gov Web site.  Now it's
			#  time to get serious and narrow down the search.
			#

if ( $token eq "TYPE" )
				{
				$capture_flag = 1 ;
				}

#
			# We want to find the<br> tokens close to the 
			# city and state, everything else is junk. If
			# we find one we count it so we can keep track of
			# how many we find.  If we find 2 it's time to
			# start looking for the city and state.

if ( $token eq "<br>" && $capture_flag ne 0 )
				{
				++$break_count ;
				# $token_string .= $token ;
				}

# 
			# We found 2<br> tokens and this means it's 
			# time to get serious about looking for city
			# and state .
			#

if ( $break_count eq 2 )
				{
				$load_array_flag = 1 ;
				}

#
			# We let the loop go once more around before we start
			# capturing the city and state into a string called 
			# $city_state_string because we don't want the <br>
			# token in our string but the next tokens which are 
			# the city and state.  The load_array_flag is now 
			# incremented again, this time to the number 2 and we 
			# check to make sure we haven't gone too far.
			#
			# If we find the word "ACCEPTABLE" we know we've gone
			# too far.  If everything is good to go we concatenate
			# tokens into the city_state_string, and separate them 
			# with a colon ":" .  We'll separate them later before
			# printing them out.
			#

if ( $load_array_flag eq 2 && $token ne "ACCEPTABLE" )
				{
				$city_state_string .= "$token:" ;
				}

# 
			# Once we've found our second "<br>" we know we've 
			# found the city and state and need look no further.
			# We set the $load_array_flag to 2 and allow the 
			# city_state_string to be built above.
			#

if ( $break_count eq 2 )
				{
				$break_count  = 0 ;
				$capture_flag = 0 ;
				$token_string = "" ;
				++$load_array_flag ;
				}

#
			# You know you've gone far enough when the word
			# "ACCEPTABLE" is found, no need to go any further.
			# We break out of the search and begin final 
			# the final processing of the city_state_string.
			#

if ( $token eq "ACCEPTABLE" )
				{
				$capture_flag    = 0 ;
				$load_array_flag = 0 ;
				break;
				}

}

#
		# Once we break out of the search, we turn the string we
		# created into an array. We want to make sure we always
		# find the state, and allow for cities with names like
		# FORT LAUDERDALE or BOCA RATON OR NEW YORK.  We also 
		# want to count how many words are in the array so we
		# can find the state and separate it from the city.
		#

my @city_state =  split /:/, $city_state_string ;

foreach $cs ( @city_state )
			{
			$cs_count++ ;
			}

--$cs_count ;
		my $city="";
		my $state="" ;
		
		#
		# Here we separate the city from the state by walking
		# backwards through the array.  The state always comes
		# first when you walk backwards so the remainder must 
		# be the city.
		#

for ( $i=0 ; $i <= $cs_count ; ++$i )
			{
			if ( $city_state[$i] )
				{
				if ( $i eq $cs_count )
					{
					$state = $city_state[$i];
					for ( $j = 0 ; $j < $i ; ++$j )
						{
						$city .= "$city_state[$j] " ;
						}
					}
				}
			}

#
		# Now that the city and state have been separated we send them
		# back to the calling function as a string, separated once again
		# with a colon that can be separated with the split command.
		#

my $city_and_state = "$city:$state" ;

return $city_and_state ;

}
	else
		{
		return "NULL" ;
		}
}
1;

################################################################################

sub post_to_server {

#
# Set these to the Web site you wish to go to.
#

$post_path   = "http://www.usps.com/cgi-bin/zip4/ctystzip2" ;
$request_str ="ctystzip=$zip_code" ;

my $ua = new LWP::UserAgent;

my $h  = new HTTP::Headers;
$h->header(Content_Type => 'application/x-www-form-urlencoded');

my $r  = new HTTP::Request("POST", $post_path, $h, $request_str);

my $response = $ua->request($r);

return $response->content;

}
1;

The most important function is the post_to_server routine. In here you need to set two variables to the correct URL where you wish to LWP-surf to.

The first variable, $post_path , holds the basic URL of the Web site and the CGI program you wish to run, but without the corresponding values.

The second variable, $request_str , holds the values that the CGI program would expect. In our case of the USPS program I went to the site, viewed the source code of the form used, and looked at the variables used in the form. You need to do this for your own uses, and also make sure to set the Request to POST or GET depending on what the receiving program is expecting.

At the end of the post_to_server routine you'll see the return command used to send the content back to the calling function.

After we get all that HTML back from the USPS Web site, we have to ignore most of it. The rest of the program scans through the HTML, skipping through until it finds the city and state, and then prints out the city, state, and zip code for a given zip code.

Sample Output
Here is a sample of the output from the program:

GIBSONBURG |OH|43431|
GRAYTOWN |OH|43432|
GYPSUM |OH|43433|
HARBOR VIEW |OH|43434|
HELENA |OH|43435|
ISLE SAINT GEORGE |OH|43436|
JERRY CITY |OH|43437|
KELLEYS ISLAND |OH|43438|
LACARNE |OH|43439|
LAKESIDE MARBLEHEAD |OH|43440|

The data is pipe-delimited, meaning that a pipe character separates the fields, which can then be loaded into a table in a database.

You'll notice that the Perl program did a good job of separating cities from states, allowing city names that have more than one word in them. You will also notice that the Perl program goes after only the first ACCEPTABLE town with the (DEFAULT) next to it. There may be other towns that use the same zip code, but for most situations you know you can use the DEFAULT city as the name of the town for a given zip code. The Post Office is going to go by the DEFAULT no matter what, so we have a great way of getting zip codes useful for our purposes.

A classic use of a zip code database would be to allow a Web site that takes memberships to validate a zip code if the customer lives in the US. Or you could save the customer the need to enter their city and state if they live in the US, and add it automatically after they sign up. I have found other countries like Australia offer their own zip code databases available for download, saving you time instead of having to create a program like this one.

For more information on how to use LWP head over to the popular Perl Web sites and dig in. For now though you should be able to use this program as a guide, and get started right away.

Another fine example of using LWP would be to set up a MySQL server behind a firewall, and have a server outside the firewall send and retrieve data from the MySQL database behind the firewall. There are many ISP's that do not have MySQL available for your Web site, but if you know the IP of your own MySQL server, say at home or elsewhere, you can set up your webserver to use LWP to send and receive information from your own MySQL box somewhere else. LWP!! If anyone is interested in Perl code for this let me know, I'll write about it in the next issue.

Printer Friendly Version

Submit Questions or Comments to Klixxx

* Due to some elements of the Perl code contained in this article,
this article is not W3C standards compliant. See W3C check below
for details on non-compliant code and W3C recommended fixes.

Klixxx Magazine Dot Net

Klixxx Dot Net

Klixxx Publishing, LLC © Copyright 2000-2008, All Rights Reserved.
The Klixxx Properties - Proudly Hosted by Cave Creek
Klixxx.com is Labeled with ICRA and Labeled with RTA
We Support the Fight Against Child Pornography @ ASACP.org

RTA Labelled

ASACP

Klixxx Recommends the Alexa Toolbar and the Google Toolbar.
Add the Digital Point Search Feature to Your Site

Klixxx Forums Community Message Board