wiredfool

Archive for the 'Old Code' Category

Barcode Recognition

Every so often, I get on to a kick where I start reading up on lisp, erlang and others that have functional, reliable, metaprogrammable, or other ‘able’ traits that indicate a more powerful language. In a way, this is partly responsible for my investigation of ruby on rails, as a metaprogramming language that has one dominant web framework that seems to do things right.

Some of this is because I see people using these tools to create software that is elegant, effective, and done with few resources. The most recent iteration is dabbledb. This is software that I wish that I’d written. I’m pretty sure that it’s written in smalltalk by a small team, who did the work between consulting jobs.

Some of this is because I want things to get easier. However, I’m reminded of Lance Armstrong: “It doesn’t get any easier, you just go faster.” The programming is still going be ‘hard’, but it better be solving bigger problems.

I’ve had a problem at work that I’ve thought about, on and off, for a year or so. We drive several check scanners, one of them with a barcode recognizer. Helpfully enough, that’s the lowest volume one — generally the scanner that one wants to upgrade from. But for people who need that barcode scanning to match a payment coupon to their database, it’s kind of an important feature. There are a few open source packages out there that do barcode recognition, some as part of an ocr engine, some as stand alone. Lots of payware activex stuff, com objects, or other closed source windows stuff. But nothing that gave a quick and easy overview that’s incorporatable or reimplementable. So I need a barcode recognizer from a black and white image. I don’t need to find the barcode, since it’s in the same place every time.

code 39
So, Code 39 barcode. There are a lot of barcode symbologies. This is a little different than the upc symbol, it’s an early attempt that’s had wide use, it’s not especially dense, but it’s easy to read.

Each ‘character’ has 9 bars, starting and ending with black, and a narrow white bar between character. 3 of those bars are wide, generally 3x wider than the narrow bars. This means that each character is a fixed width, probably about 16x the smallest unit (3×3 + 7×1). Also, there’s supposed to be a large white space at the beginning and end of the barcode, and start and stop characters at each end. So the total number of bars, white and black, including start/stop codes, but not the end buffer space, will be (chars)*10 + 19 bars.

So now, for some functional goodness. The essential trick here is to encode a string of bits from the image in run length encoding, here represented by tuples of (length, color) in an array. Once you have that, you can figure out if you have a reasonable number of bars (one per run length entry), characters, and from that, the barcode. Everything in the problem can be modeled as a list, and all (save one) of the operations can be a map or filter on the list.

I’m doing this in python, usng the itertools package and a curry implementation that gives me partial function application. (e.g. bar = curry(foo,a), bar(b) == foo(a,b)) Itertools gets me the groupby function, which returns lists of identical(ish) items. Curry, well, curry gets me partial function evaluation, which is really some syntatcic sugar to let me use map instead of an explicit for loop.

First up, we need a row of pixels. I’m assuming that we know where the bar code is in the image, and we just need to interpret it. Finding the barcode is a problem left for the reader. We’re using the python imaging library, so extracting a row is a quick function. Here I’m cropping the barcode region to a 1 pixel high area at a height v in the region, then getting the data from the image.

	def extract_row(self, img, v):
		(w,h) = img.size
		return img.crop((0,v,w,v+1)).getdata()

For a black and white image, this comes back as a list of integers, either 0 (black) or 255 (white).

Next, we need to take this pixel data and get something that approximates bars. Run length encoding does the trick here, since we would like color and width for each item on our scan line. This is the first use of the itertools.groupby function, which returns a value and an iterator of the items for each identical value. I’m grabbing the length of the value list and the value, mapping the value to a color, and returning it as a tuple, then chopping off the whitespace at each end.

	def to_rle(self, row):		
		mp = {0: 'b', 255: 'w'}
		return [(len(list(g)), mp[k]) for k,g in itertools.groupby(row)][1:-1]

This should return a list of [(len, ‘b’), (len, ‘w’), (len,’b’) …] where the first and last items are black. If they’re not, then we should just discard the line. It’s much easier to just discard lines that don’t make sense, either from a bad scan or missing barcode information than to try to tough it up. There are always more scanlines to try. (Until there aren’t, and then it may be worth some touchup).

Now, we have a list of white an black regions, we need to determine if they are wide or narrow bars, then split them into individual characters. Since there is a rough correspondence between the pixel width of the barcode and the number of narrow bar widths, I’ve set a threshold of 2x the narrow bar width as the divider for wide/narrow. We can calculate the narrow bar spacing by pxLen/(16*characters), and # chars = (len(rle)+1)/10. Or, in code:

	def threshold(self, rle):
		n = (len(rle)+1)/10
		pxlen = sum(map(lambda x: x[0], rle))		
		return 2*(pxlen / (16*n))

And apply it using:

	def to_bars(self, rle):
		return map(curry(lambda x,y: str(int(y[0] > x)), self.sym.threshold(rle)), rle)

In this case, I’m returning it as ‘1’, and ‘0’ for wide and narrow, respectively. I could return bits or booleans or strings, but I found the 1 and 0 easy enough for the character substitution. In the future, I’ll probably make them binary and try to specify the mappings as character set transformations. But not today.

The penultimate step is to chunk into characters, another use of the groupby function, this time with a little stored state. I want to pull off 10 bars at a time (9 + whitespace), so I’m using a helper that will give me a integer div 10 of the number of times that it’s been called.

	def _iterkey(self, val):
		self._iterstate +=1
		return (self._iterstate-1) / 10
	
	def chunk(self, bars):
		self._iterstate = 0
		return [''.join(list(g)[:9]) for k,g in itertools.groupby(bars, self._iterkey)]

Finally, we need to turn the chunks into characters. I’ve got a map of the 9 character chunks to the character that they represent, processed from the wikipedia and other documentation above. So it’s a simple matter of subbing into the map and dropping the start and stop characters:

	def to_chars(self, chunked):
		return ''.join(map(lambda x: self.sym.bkw[x], chunked)[1:-1])

Putting this all together with some error checking, retries on the next scan line, we get:

 	def recognize(self, img):
		(w,h) = img.size
		for scan in range(1,h-1):
			rle = self.to_rle(self.extract_row(img, scan))			
			if len(rle) < self.sym.min_length : continue			
			if not self.sym.check_len(rle): continue
			chunks = self.sym.chunk(self.to_bars(rle))
			if len(self.sym.invalid_chunks(chunks)) : continue
			try:
				ret = self.sym.extract(self.to_chars(chunks))
				if len(ret): return ret
			except:
				continue
		return None

On my linux (ubuntu 6.06, amd64, 3600?) box, this does about one barcode per .01 sec, where the barcodes are about 280x100px, extracted from about 1 megapixel images. On average, I'm trying about 40 scanlines before I hit on one that's error free. It's about 2.5x slower on the mac (macbook core duo) for reasons that I haven't figured out yet.

Link to the full source.
Link to an archive with the code, test code and a sample image.

2 comments

Updated Imagemagick Thumbnail script

A few years ago, I posted a script that I used for generating image thumbnails. I’ve changed my usage of it, so I’m posting another go of it. Since then, OSX has shipped, the Imagemagick .pkg distribution comes with the perl bindings, and I’ve figured out that I generally just want three sizes and 90 degree rotations for the images.

So I generate three sizes into a seperate directory from (usually) 1600px max size images: a 175px (thumbnail, -tm), 400px (medium, -md) and 800px (penultimate, -pt). There’s also a straight 90 degree rotation for each size that gets a -r in the filename. There are also a few index.html files for that quick web based overview.

Call the script using

perl thumbnail.pl directory

This script assumes that your source images are in ~/Pictures/directory/, and that the output should be put in ~/Sites/thumbs/directory/. Edit the paths if you want them in different locations.

Download thumbnail.pl.

No comments

Updated Imagemagick Thumbnail script

A few years ago, I posted a script that I used for generating image thumbnails. I’ve changed my usage of it, so I’m posting another go of it. Since then, OSX has shipped, the Imagemagick .pkg distribution comes with the perl bindings, and I’ve figured out that I generally just want three sizes and 90 degree rotations for the images.

So I generate three sizes into a seperate directory from (usually) 1600px max size images: a 175px (thumbnail, -tm), 400px (medium, -md) and 800px (penultimate, -pt). There’s also a straight 90 degree rotation for each size that gets a -r in the filename. There are also a few index.html files for that quick web based overview.

Call the script using perl thumbnail.pl [directory]

This script assumes that your source images are in ~/Pictures/[directory]/, and that the output should be put in ~/Sites/thumbs/[directory]/. Edit the paths if you want them in different locations.

#! /usr/bin/perl
 
use Image::Magick;
 
$targetExtension = "(jpg)|(tif)|(bmp)|(JPG)";
$thumbnailExtension = "jpg";
$thumbnailSize = "175x175";
$mediumSize = "400x400";
$thumbnailName = "-tm";
$pentSize="800x800";
$mediumName = "-md";
$lgName = "-lg";
$pentName = "-pt";
 
($startDir, @rest) = @ARGV;
 
my $image;
 
$destDir = $ENV{HOME}."/Sites/thumbs/".$startDir;
 
if  (!( $startDir =~ /\//)){
    $startDir = $ENV{HOME}."/Pictures/".$startDir;
    print $startDir, "\n";
}
 
opendir (START, $startDir) || die "Couldn't Open Start dir: $startDir";
 
@files = readdir(START);
 
closedir (START);
 
if (not -d $destDir) {
        mkdir ($destDir, 0775);
}
 
open (OUTFILE, ">$destDir/index.html") or die "Couldn't open index.html";
open (OUTFILE2, ">$destDir/index2.html") or die "Couldn't open index2.html";
open (OUTFILE3, ">$destDir/index3.html") or die "Couldn't open index3.html";
 
foreach $file (sort @files) {
        ($fname,$extension) = split(/\./,$file);
        $fname =~ s/$lgName//;
        if ($extension =~ /$targetExtension/i) {
                print "$fname \n";
                $image = new Image::Magick;
                $image->Read("$startDir/$file");
                $image->Scale(geometry=>$pentSize);
                $image->Write("$destDir/$fname$pentName.$thumbnailExtension");
                $image->Rotate(degrees=>90);
                $image->Write("$destDir/$fname$pentName-r.$thumbnailExtension");
 
                $image->Scale(geometry=>$mediumSize);
                $image->Write("$destDir/$fname$mediumName-r.$thumbnailExtension");
                $image->Rotate(degrees=>270);
                $image->Write("$destDir/$fname$mediumName.$thumbnailExtension");
 
                $image->Scale(geometry=>$thumbnailSize);
                $image->Write("$destDir/$fname$thumbnailName.$thumbnailExtension");
                $image->Rotate(degrees=>90);
                $image->Write("$destDir/$fname$thumbnailName-r.$thumbnailExtension");
 
        print OUTFILE "< \a href=\"$fname$pentName.$thumbnailExtension\">\
n";
        print OUTFILE2 "< \a href=\"$fname$pentName-r.$thumbnailExtension\">
\n";
        print OUTFILE3 "< \a href=\"$fname$pentName.$thumbnailExtension\">< /a>\n";
        }
}

close OUTFILE;
close OUTFILE2;
close OUTFILE3;
No comments

Updated Imagemagick Thumbnail script

A few years ago, I posted a script that I used for generating image thumbnails. I’ve changed my usage of it, so I’m posting another go of it. Since then, OSX has shipped, the Imagemagick .pkg distribution comes with the perl bindings, and I’ve figured out that I generally just want three sizes and 90 degree rotations for the images.

So I generate three sizes into a seperate directory from (usually) 1600px max size images: a 175px (thumbnail, -tm), 400px (medium, -md) and 800px (penultimate, -pt). There’s also a straight 90 degree rotation for each size that gets a -r in the filename. There are also a few index.html files for that quick web based overview.

Call the script using perl thumbnail.pl [directory]

This script assumes that your source images are in ~/Pictures/[directory]/, and that the output should be put in ~/Sites/thumbs/[directory]/. Edit the paths if you want them in different locations.

#! /usr/bin/perl
 
use Image::Magick;
 
$targetExtension = "(jpg)|(tif)|(bmp)|(JPG)";
$thumbnailExtension = "jpg";
$thumbnailSize = "175x175";
$mediumSize = "400x400";
$thumbnailName = "-tm";
$pentSize="800x800";
$mediumName = "-md";
$lgName = "-lg";
$pentName = "-pt";
 
($startDir, @rest) = @ARGV;
 
my $image;
 
$destDir = $ENV{HOME}."/Sites/thumbs/".$startDir;
 
if  (!( $startDir =~ /\//)){
    $startDir = $ENV{HOME}."/Pictures/".$startDir;
    print $startDir, "\n";
}
 
opendir (START, $startDir) || die "Couldn't Open Start dir: $startDir";
 
@files = readdir(START);
 
closedir (START);
 
if (not -d $destDir) {
        mkdir ($destDir, 0775);
}
 
open (OUTFILE, ">$destDir/index.html") or die "Couldn't open index.html";
open (OUTFILE2, ">$destDir/index2.html") or die "Couldn't open index2.html";
open (OUTFILE3, ">$destDir/index3.html") or die "Couldn't open index3.html";
 
foreach $file (sort @files) {
        ($fname,$extension) = split(/\./,$file);
        $fname =~ s/$lgName//;
        if ($extension =~ /$targetExtension/i) {
                print "$fname \n";
                $image = new Image::Magick;
                $image->Read("$startDir/$file");
                $image->Scale(geometry=>$pentSize);
                $image->Write("$destDir/$fname$pentName.$thumbnailExtension");
                $image->Rotate(degrees=>90);
                $image->Write("$destDir/$fname$pentName-r.$thumbnailExtension");
 
                $image->Scale(geometry=>$mediumSize);
                $image->Write("$destDir/$fname$mediumName-r.$thumbnailExtension");
                $image->Rotate(degrees=>270);
                $image->Write("$destDir/$fname$mediumName.$thumbnailExtension");
 
                $image->Scale(geometry=>$thumbnailSize);
                $image->Write("$destDir/$fname$thumbnailName.$thumbnailExtension");
                $image->Rotate(degrees=>90);
                $image->Write("$destDir/$fname$thumbnailName-r.$thumbnailExtension");
 
        print OUTFILE "\n";
        print OUTFILE2 "\n";
        print OUTFILE3 "< /a>\n";
        }
}

close OUTFILE;
close OUTFILE2;
close OUTFILE3;
No comments

TOC Updates

Version 0.21

  • 5/10 Fixed encoding error that f**ed up html.

Version 0.2

  • 5/10 Added a parsing class so that callbacks can be written using logical parameters, rather than the “data” string.
  • 5/10 Added logging levels and took out some of the logging of expected occurances.
  • 5/10 Callbacks are now called in a threaded fashion.
No comments

Patch Tool

Version: 0.16, 9/5/2002

Download: http://radio.weblogs.com/0001179/gems/patch.root
To Install: Put in your Tools folder in Radio Userland or Frontier.
License: BSD. Copyright Eric Soroos, 2002. Released with the permission of SocialEcology Inc.

***What it does

This is a developer tool that provides patch, diff, and cvs integration services for Usertalk scripts in Frontier and Radio Userland. Its integration level is currently roughly that of app glue. There are plans to use this core to provide comprehensive source management.

Starting with release 0.16, there is code to dump the contents of a (nested) database table to disk in text format and to read it back in again. The files are written in a way that allows interoperation with conventional source code management systems (e.g. CVS). This could also be suitable for backup of ueser preference settings. Scripts and outlines are written in opml format, binaries and menubars are written using base64 encoding, everything else is written as plain text.

If you are running on Mac OSX, it will do the diffs and patches on your local machine. If you are not, it will contact a webservice running on differenceEngine.wiredfool.com which will perform the differences for you.

***How to Use

There are two entry points for the difference code:

  • diffString = patchSuite.getDiff(@original, @modified)

    GetDiff will return a context diff of the xml representation of the script. You will probably want to edit the diffString to not include spurious differences in the header properties.

  • errString = patchSuite.applyDiff(@original, @patch, @patchedScript)

    ApplyDiff takes the context diff created in step 1 and applies it to the original script, then places the result in a new script and compiles it. As far as I know, the only formatting that is lost is the outline expansion state. ErrString is the result from the command line call to patch.

There are also two entry points for the file system code:

  • patchSuite.dumpToFileSystem(@table)

    Dump to file system will prompt for a folder to save the contents of the table. It will write out the contents of the table to the file system, With the exception of items with a tab in the address. This excludes the compiled versions of xml items.

  • patchSuite.readFromFileSystem(@destination, flIgnoreCVSLint=true)

    ReadFromFileSystem takes the items in the file system and assembles them into the table at the destination address. The one optional parameter ignores all directories named CVS, which are status repositories for the CVS system and not applicable to Frontier.

***Tweakable Bits

You can change the rpc server at patchData.prefs.rpcServer if you don’t want to use mine.

***Known Issues 9/5/2002

  • (Dump) There should probably be a flag to ignore the tab characters in addresses.
  • (Dump/read) Has been tested well on script, outline, string, boolean, and number types. It chas been tested less well on lists, binaries, and menus (but should still work). It loses formatting on wptexts. It should work on most other types, but the exotic ones have not really been tested. There are probably going to be issues with items in lists where the item length is > 256 characters and anywhere else that [itemTypeCast](string(item)) isn’t an identity relation. Restoring addresses may require that I strip the root portion off first.
  • (Dump/Restore) Fails when trying to dump the running script. Probably will die if it overwrites the running script on a restore. Workaround: copy the script to system.temp and run from there.
No comments

PicsPicker Updates

10/25/01 – 10am
Fix for the sitename bug that Raymond Yee found.

8/12/01 – 9pm
Added a random image macro: picsPickerMacros.randomImage(strImageList, metadataField=”numRandom”, border=0). If the numRandom field is included via metadata, then the specified number of random images will be shown. If not, then one will be chosen from the list.

The Linked Image macro now will link to the bare picture if the user doesn’t have access to the discussion group and the target is a picture.

5/14/01 – 5pm
Fixed a bug where the special macros in the image gallery and next prev templates would be turned into html entities.

5/14/01 – 4pm
Fixed a bug in the imageGallery and nextPrevLinks macros where they would have errors when called from a storyreader url.

5/11/01 – 4pm
Added the picsPickerData.prefs.databases table, since the installer looks for that to add the database to user.databases.

No comments

About The PicsPicker Plugin

The PicsPicker plugin automates some of the image management tasks that I find to be time consuming on a manila site. Download from updates.wiredfool.com: .root or .sit

There are two main functions of the plugin:

Image Upload:
You can paste in multiple image urls to the text box and the server will fetch them and insert them as pictures in your manila site. I find this useful because I have a script that creates web accessible thumbnails from a folder of digital camera images. Just paste the urls and go. If you wish to edit the text or the image name later, you can edit the discussion group message.
Image Gallery:
The image gallery settings allow you to easily create an image gallery with a minimum of work. The templates for the image gallery and the next/previous macro are editable on a user by user basis. There is a Sample Gallery that shows the use and results of the imageGallery macro.
There are 3 macros that are used to make the image galleries:

  • picsPickerMacros.imageGallery(imageList, linkedImageList, border:2) – This macro uses the image gallery template to create a series of pages that display the image, the caption stored in the body of the discussion group message, and (optionally) the next previous links. The imageList is a space or comma seperated list of the image numbers in the discussion group. The linkedImageList is a list of image numbers for the full size versions of those images. (e.g. {
    picsPickerMacros.imageGallery(“10 12 14″,”11 13 15”)} ) The Linked Image List is optional.
  • picsPickerMacros.nextPrevLinks(count) – Produces next/previous links for the image gallery. Count is the number of pages. If the nextprev macro is included in the image gallery template, then count is determined automatically.
  • picsPickerMacros.randomImage(strImageList, metadataField=”numRandom”, border=0) – This macro will return an image link from the specified list of image numbers. If the numRandom field is included via metadata, then the specified number of random images will be added. If not, then one will be chosen from the list.

Installation:
Put the downloaded database in your Frontier:Guest Databases:Apps folder and open it. Run the _fInstall script. You should be ready to go. Updates are served from updates.wiredfool.com, and will be noted on the updates page.
Known Issues:
It is unclear if the image gallery parts of this plugin will work with static rendering. It is also unclear if they will work with the static image serving. I will be investigating this further after the first release.

Fine Print:
There are no rpc handlers associated with this plugin. It was written using the prefs suite wizard framework for the user interface. The basic structure was created with the plugin factory. See the updates page for information on updates. This plugin is copyright © Eric Soroos, 2001 and released for use under the Gnu Public License. Released with the permission of Socialecology Inc.

1 comment

JSP Proxy Responder

This is a responder that traps any page request ending in .jsp and proxies that request to another server.

This responder does two things to the request before it forwards it to the destination server. First, it replaces the Host header with an admin configurable value. This allows you to proxy multiple virtual domains to one configurable virtual domain on the .jsp server. It also rewrites the Connection header to make sure that the connection is closed after the request.

***Download

Download the fttb file. If it comes down as a text file in the browser, simply save as source.

***Install

Download, then open the file from within Frontier or Radio Userland. It will ask you where to install the table, it needs to be at user.webserver.responders.jspProxy.

***Configuration

Set the value at user.webserver.responder.jspProxy.requestHost to the desired host for the request. Set requestIp and requestPort to the ip address and port that your jsp server is listening to.

If you wish to proxy other types of requests, simply change the condition to one matching what you wish to proxy. (e.g., endsWith “.asp”, contains “servlet”). Just be aware that this responder catches requests before mainresponder does, so that if you map something like /discuss, you will hose a lot of functionality on your machine.

If the above two paragraphs are greek to you, either I’m not writing clearly or this software isn’t for you.

No comments

Esoteric Settings Updates

Updates are once again availiable through the normal root updates process from updates.wiredfool.com.

Latest Changes

10/25/01
Fix for the couldn’t compile this script error that Clark Venable found. This is essentially the same bug as was fixed in picsPicker.

5-31-2001 — Finally got around to removing the double title on the settings pages. It requires a disable/reenable cycle, since it changes the stub in the website.

No comments

Next Page »