Tag: tutorial

Why have regular expressions survived seven decades of technological disruption? Because coders who understand regular expressions have a massive advantage when working with textual data. They can write in a single line of code what takes others dozens!

This article is all about the re.compile(pattern) method of Python’s re library. Before we dive into re.compile(), let’s get an overview of the four related methods you must understand:

• The findall(pattern, string) method returns a list of string matches. Read more in our blog tutorial.
• The search(pattern, string) method returns a match object of the first match. Read more in our blog tutorial.
• The match(pattern, string) method returns a match object if the regex matches at the beginning of the string. Read more in our blog tutorial.
• The fullmatch(pattern, string) method returns a match object if the regex matches the whole string. Read more in our blog tutorial.

Equipped with this quick overview of the most critical regex methods, let’s answer the following question:

How Does re.compile() Work in Python?

The re.compile(pattern) method returns a regular expression object (see next section)

You then use the object to call important regex methods such as search(string), match(string), fullmatch(string), and findall(string).

In short: You compile the pattern first. You search the pattern in a string second.

This two-step approach is more efficient than calling, say, search(pattern, string) at once. That is, IF you call the search() method multiple times on the same pattern. Why? Because you can reuse the compiled pattern multiple times.

Here’s an example:

import re # These two lines ...
regex = re.compile('Py...n')
match = regex.search('Python is great') # ... are equivalent to ...
match = re.search('Py...n', 'Python is great')

In both instances, the match variable contains the following match object:

<re.Match object; span=(0, 6), match='Python'>

But in the first case, we can find the pattern not only in the string ‘Python is great’ but also in other strings—without any redundant work of compiling the pattern again and again.

Specification:

re.compile(pattern, flags=0)

The method has up to two arguments.

• pattern: the regular expression pattern that you want to match.
• flags (optional argument): a more advanced modifier that allows you to customize the behavior of the function. Want to know how to use those flags? Check out this detailed article on the Finxter blog.

We’ll explore those arguments in more detail later.

Return Value:

The re.compile(patterns, flags) method returns a regular expression object. You may ask (and rightly so):

What’s a Regular Expression Object?

Python internally creates a regular expression object (from the Pattern class) to prepare the pattern matching process. You can call the following methods on the regex object:

Method	Description
Pattern.search(string[, pos[, endpos]])	Searches the regex anywhere in the string and returns a match object or None. You can define start and end positions of the search.
Pattern.match(string[, pos[, endpos]])	Searches the regex at the beginning of the string and returns a match object or None. You can define start and end positions of the search.
Pattern.fullmatch(string[, pos[, endpos]])	Matches the regex with the whole string and returns a match object or None. You can define start and end positions of the search.
Pattern.split(string, maxsplit=0)	Divides the string into a list of substrings. The regex is the delimiter. You can define a maximum number of splits.
Pattern.findall(string[, pos[, endpos]])	Searches the regex anywhere in the string and returns a list of matching substrings. You can define start and end positions of the search.
Pattern.finditer(string[, pos[, endpos]])	Returns an iterator that goes over all matches of the regex in the string (returns one match object after another). You can define the start and end positions of the search.
Pattern.sub(repl, string, count=0)	Returns a new string by replacing the first count occurrences of the regex in the string (from left to right) with the replacement string repl.
Pattern.subn(repl, string, count=0)	Returns a new string by replacing the first count occurrences of the regex in the string (from left to right) with the replacement string repl. However, it returns a tuple with the replaced string as the first and the number of successful replacements as the second tuple value.

If you’re familiar with the most basic regex methods, you’ll realize that all of them appear in this table. But there’s one distinction: you don’t have to define the pattern as an argument. For example, the regex method re.search(pattern, string) will internally compile a regex object p and then call p.search(string).

You can see this fact in the official implementation of the re.search(pattern, string) method:

def search(pattern, string, flags=0): """Scan through string looking for a match to the pattern, returning a Match object, or None if no match was found.""" return _compile(pattern, flags).search(string)

(Source: GitHub repository of the re package)

The re.search(pattern, string) method is a mere wrapper for compiling the pattern first and calling the p.search(string) function on the compiled regex object p.

Is It Worth Using Python’s re.compile()?

No, in the vast majority of cases, it’s not worth the extra line.

Consider the following example:

import re # These two lines ...
regex = re.compile('Py...n')
match = regex.search('Python is great') # ... are equivalent to ...
match = re.search('Py...n', 'Python is great')

Don’t get me wrong. Compiling a pattern once and using it many times throughout your code (e.g., in a loop) comes with a big performance benefit. In some anecdotal cases, compiling the pattern first lead to 10x to 50x speedup compared to compiling it again and again.

But the reason it is not worth the extra line is that Python’s re library ships with an internal cache. At the time of this writing, the cache has a limit of up to 512 compiled regex objects. So for the first 512 times, you can be sure when calling re.search(pattern, string) that the cache contains the compiled pattern already.

Here’s the relevant code snippet from re’s GitHub repository:

# --------------------------------------------------------------------
# internals _cache = {} # ordered! _MAXCACHE = 512
def _compile(pattern, flags): # internal: compile pattern if isinstance(flags, RegexFlag): flags = flags.value try: return _cache[type(pattern), pattern, flags] except KeyError: pass if isinstance(pattern, Pattern): if flags: raise ValueError( "cannot process flags argument with a compiled pattern") return pattern if not sre_compile.isstring(pattern): raise TypeError("first argument must be string or compiled pattern") p = sre_compile.compile(pattern, flags) if not (flags & DEBUG): if len(_cache) >= _MAXCACHE: # Drop the oldest item try: del _cache[next(iter(_cache))] except (StopIteration, RuntimeError, KeyError): pass _cache[type(pattern), pattern, flags] = p return p

Can you find the spots where the cache is initialized and used?

While in most cases, you don’t need to compile a pattern, in some cases, you should. These follow directly from the previous implementation:

• You’ve got more than MAXCACHE patterns in your code.
• You’ve got more than MAXCACHE different patterns between two same pattern instances. Only in this case, you will see “cache misses” where the cache has already flushed the seemingly stale pattern instances to make room for newer ones.
• You reuse the pattern multiple times. Because if you don’t, it won’t make sense to use sparse memory to save them in your memory.
• (Even then, it may only be useful if the patterns are relatively complicated. Otherwise, you won’t see a lot of performance benefits in practice.)

To summarize, compiling the pattern first and storing the compiled pattern in a variable for later use is often nothing but “premature optimization”—one of the deadly sins of beginner and intermediate programmers.

What Does re.compile() Really Do?

It doesn’t seem like a lot, does it? My intuition was that the real work is in finding the pattern in the text—which happens after compilation. And, of course, matching the pattern is the hard part. But a sensible compilation helps a lot in preparing the pattern to be matched efficiently by the regex engine—work that would otherwise have be done by the regex engine.

Regex’s compile() method does a lot of things such as:

• Combine two subsequent characters in the regex if they together indicate a special symbol such as certain Greek symbols.
• Prepare the regex to ignore uppercase and lowercase.
• Check for certain (smaller) patterns in the regex.
• Analyze matching groups in the regex enclosed in parentheses.

Here’s the implemenation of the compile() method—it looks more complicated than expected, no?

def _compile(code, pattern, flags): # internal: compile a (sub)pattern emit = code.append _len = len LITERAL_CODES = _LITERAL_CODES REPEATING_CODES = _REPEATING_CODES SUCCESS_CODES = _SUCCESS_CODES ASSERT_CODES = _ASSERT_CODES iscased = None tolower = None fixes = None if flags & SRE_FLAG_IGNORECASE and not flags & SRE_FLAG_LOCALE: if flags & SRE_FLAG_UNICODE: iscased = _sre.unicode_iscased tolower = _sre.unicode_tolower fixes = _ignorecase_fixes else: iscased = _sre.ascii_iscased tolower = _sre.ascii_tolower for op, av in pattern: if op in LITERAL_CODES: if not flags & SRE_FLAG_IGNORECASE: emit(op) emit(av) elif flags & SRE_FLAG_LOCALE: emit(OP_LOCALE_IGNORE[op]) emit(av) elif not iscased(av): emit(op) emit(av) else: lo = tolower(av) if not fixes: # ascii emit(OP_IGNORE[op]) emit(lo) elif lo not in fixes: emit(OP_UNICODE_IGNORE[op]) emit(lo) else: emit(IN_UNI_IGNORE) skip = _len(code); emit(0) if op is NOT_LITERAL: emit(NEGATE) for k in (lo,) + fixes[lo]: emit(LITERAL) emit(k) emit(FAILURE) code[skip] = _len(code) - skip elif op is IN: charset, hascased = _optimize_charset(av, iscased, tolower, fixes) if flags & SRE_FLAG_IGNORECASE and flags & SRE_FLAG_LOCALE: emit(IN_LOC_IGNORE) elif not hascased: emit(IN) elif not fixes: # ascii emit(IN_IGNORE) else: emit(IN_UNI_IGNORE) skip = _len(code); emit(0) _compile_charset(charset, flags, code) code[skip] = _len(code) - skip elif op is ANY: if flags & SRE_FLAG_DOTALL: emit(ANY_ALL) else: emit(ANY) elif op in REPEATING_CODES: if flags & SRE_FLAG_TEMPLATE: raise error("internal: unsupported template operator %r" % (op,)) if _simple(av[2]): if op is MAX_REPEAT: emit(REPEAT_ONE) else: emit(MIN_REPEAT_ONE) skip = _len(code); emit(0) emit(av[0]) emit(av[1]) _compile(code, av[2], flags) emit(SUCCESS) code[skip] = _len(code) - skip else: emit(REPEAT) skip = _len(code); emit(0) emit(av[0]) emit(av[1]) _compile(code, av[2], flags) code[skip] = _len(code) - skip if op is MAX_REPEAT: emit(MAX_UNTIL) else: emit(MIN_UNTIL) elif op is SUBPATTERN: group, add_flags, del_flags, p = av if group: emit(MARK) emit((group-1)*2) # _compile_info(code, p, _combine_flags(flags, add_flags, del_flags)) _compile(code, p, _combine_flags(flags, add_flags, del_flags)) if group: emit(MARK) emit((group-1)*2+1) elif op in SUCCESS_CODES: emit(op) elif op in ASSERT_CODES: emit(op) skip = _len(code); emit(0) if av[0] >= 0: emit(0) # look ahead else: lo, hi = av[1].getwidth() if lo != hi: raise error("look-behind requires fixed-width pattern") emit(lo) # look behind _compile(code, av[1], flags) emit(SUCCESS) code[skip] = _len(code) - skip elif op is CALL: emit(op) skip = _len(code); emit(0) _compile(code, av, flags) emit(SUCCESS) code[skip] = _len(code) - skip elif op is AT: emit(op) if flags & SRE_FLAG_MULTILINE: av = AT_MULTILINE.get(av, av) if flags & SRE_FLAG_LOCALE: av = AT_LOCALE.get(av, av) elif flags & SRE_FLAG_UNICODE: av = AT_UNICODE.get(av, av) emit(av) elif op is BRANCH: emit(op) tail = [] tailappend = tail.append for av in av[1]: skip = _len(code); emit(0) # _compile_info(code, av, flags) _compile(code, av, flags) emit(JUMP) tailappend(_len(code)); emit(0) code[skip] = _len(code) - skip emit(FAILURE) # end of branch for tail in tail: code[tail] = _len(code) - tail elif op is CATEGORY: emit(op) if flags & SRE_FLAG_LOCALE: av = CH_LOCALE[av] elif flags & SRE_FLAG_UNICODE: av = CH_UNICODE[av] emit(av) elif op is GROUPREF: if not flags & SRE_FLAG_IGNORECASE: emit(op) elif flags & SRE_FLAG_LOCALE: emit(GROUPREF_LOC_IGNORE) elif not fixes: # ascii emit(GROUPREF_IGNORE) else: emit(GROUPREF_UNI_IGNORE) emit(av-1) elif op is GROUPREF_EXISTS: emit(op) emit(av[0]-1) skipyes = _len(code); emit(0) _compile(code, av[1], flags) if av[2]: emit(JUMP) skipno = _len(code); emit(0) code[skipyes] = _len(code) - skipyes + 1 _compile(code, av[2], flags) code[skipno] = _len(code) - skipno else: code[skipyes] = _len(code) - skipyes + 1 else: raise error("internal: unsupported operand type %r" % (op,))

Don’t worry, you don’t need to understand the code. Just note that all this work would have to be done by the regex engine at “matching runtime” if you wouldn’t compile the pattern first. If we can do it only once, it’s certainly a low-hanging fruit for performance optimizations—especially for long regular expression patterns.

How to Use the Optional Flag Argument?

As you’ve seen in the specification, the compile() method comes with an optional third ‘flag’ argument:

re.compile(pattern, flags=0)

What’s the purpose of the flags argument?

Flags allow you to control the regular expression engine. Because regular expressions are so powerful, they are a useful way of switching on and off certain features (for example, whether to ignore capitalization when matching your regex).

Syntax	Meaning
re.ASCII	If you don’t use this flag, the special Python regex symbols w, W, b, B, d, D, s and S will match Unicode characters. If you use this flag, those special symbols will match only ASCII characters — as the name suggests.
re.A	Same as re.ASCII
re.DEBUG	If you use this flag, Python will print some useful information to the shell that helps you debugging your regex.
re.IGNORECASE	If you use this flag, the regex engine will perform case-insensitive matching. So if you’re searching for [A-Z], it will also match [a-z].
re.I	Same as re.IGNORECASE
re.LOCALE	Don’t use this flag — ever. It’s depreciated—the idea was to perform case-insensitive matching depending on your current locale. But it isn’t reliable.
re.L	Same as re.LOCALE
re.MULTILINE	This flag switches on the following feature: the start-of-the-string regex ‘^’ matches at the beginning of each line (rather than only at the beginning of the string). The same holds for the end-of-the-string regex ‘$’ that now matches also at the end of each line in a multi-line string.
re.M	Same as re.MULTILINE
re.DOTALL	Without using this flag, the dot regex ‘.’ matches all characters except the newline character ‘n’. Switch on this flag to really match all characters including the newline character.
re.S	Same as re.DOTALL
re.VERBOSE	To improve the readability of complicated regular expressions, you may want to allow comments and (multi-line) formatting of the regex itself. This is possible with this flag: all whitespace characters and lines that start with the character ‘#’ are ignored in the regex.
re.X	Same as re.VERBOSE

Here’s how you’d use it in a practical example:

import re text = 'Python is great (python really is)' regex = re.compile('Py...n', flags=re.IGNORECASE) matches = regex.findall(text)
print(matches)
# ['Python', 'python']

Although your regex ‘Python’ is uppercase, we ignore the capitalization by using the flag re.IGNORECASE.

Where to Go From Here?

You’ve learned about the re.compile(pattern) method that prepares the regular expression pattern—and returns a regex object which you can use multiple times in your code.

Learning Python is hard. But if you cheat, it isn’t as hard as it has to be:

Download 8 Free Python Cheat Sheets now!

Why have regular expressions survived seven decades of technological disruption? Because coders who understand regular expressions have a massive advantage when working with textual data. They can write in a single line of code what takes others dozens!

This article is all about the re.fullmatch(pattern, string) method of Python’s re library. There are three similar methods to help you use regular expressions:

• The findall(pattern, string) method returns a list of string matches. Check out our blog tutorial.
• The search(pattern, string) method returns a match object of the first match. Check out our blog tutorial.
• The match(pattern, string) method returns a match object if the regex matches at the beginning of the string. Check out our blog tutorial.

So how does the re.fullmatch() method work? Let’s study the specification.

How Does re.fullmatch() Work in Python?

The re.fullmatch(pattern, string) method returns a match object if the pattern matches the whole string.

Specification:

re.fullmatch(pattern, string, flags=0)

The re.fullmatch() method has up to three arguments.

• pattern: the regular expression pattern that you want to match.
• string: the string which you want to search for the pattern.
• flags (optional argument): a more advanced modifier that allows you to customize the behavior of the function. Want to know how to use those flags? Check out this detailed article on the Finxter blog.

We’ll explore them in more detail later.

Return Value:

The re.fullmatch() method returns a match object. You may ask (and rightly so):

What’s a Match Object?

If a regular expression matches a part of your string, there’s a lot of useful information that comes with it: what’s the exact position of the match? Which regex groups were matched—and where?

The match object is a simple wrapper for this information. Some regex methods of the re package in Python—such as fullmatch()—automatically create a match object upon the first pattern match.

At this point, you don’t need to explore the match object in detail. Just know that we can access the start and end positions of the match in the string by calling the methods m.start() and m.end() on the match object m:

>>> m = re.fullmatch('h...o', 'hello')
>>> m.start()
0
>>> m.end()
5

In the first line, you create a match object m by using the re.fullmatch() method. The pattern ‘h…o’ matches in the string ‘hello’ at start position 0 and end position 5. But note that as the fullmatch() method always attempts to match the whole string, the m.start() method will always return zero.

Now, you know the purpose of the match object in Python. Let’s check out a few examples of re.fullmatch()!

A Guided Example for re.fullmatch()

First, you import the re module and create the text string to be searched for the regex patterns:

>>> import re
>>> text = '''
Call me Ishmael. Some years ago--never mind how long precisely
--having little or no money in my purse, and nothing particular
to interest me on shore, I thought I would sail about a little
and see the watery part of the world. '''

Let’s say you want to match the full text with this regular expression:

>>> re.fullmatch('Call(.|\n)*', text)
>>> 

The first argument is the pattern to be found: 'Call(.|\n)*'. The second argument is the text to be analyzed. You stored the multi-line string in the variable text—so you take this as the second argument. The third argument flags of the fullmatch() method is optional and we skip it in the code.

There’s no output! This means that the re.fullmatch() method did not return a match object. Why? Because at the beginning of the string, there’s no match for the ‘Call’ part of the regex. The regex starts with an empty line!

So how can we fix this? Simple, by matching a new line character ‘\n’ at the beginning of the string.

>>> re.fullmatch('\nCall(.|\n)*', text)
<re.Match object; span=(0, 229), match='\nCall me Ishmael. Some years ago--never mind how>

The regex (.|\n)* matches an arbitrary number of characters (new line characters or not) after the prefix ‘\nCall’. This matches the whole text so the result is a match object. Note that there are 229 matching positions so the string included in resulting match object is only the prefix of the whole matching string. This fact is often overlooked by beginner coders.

What’s the Difference Between re.fullmatch() and re.match()?

The methods re.fullmatch() and re.match(pattern, string) both return a match object. Both attempt to match at the beginning of the string. The only difference is that re.fullmatch() also attempts to match the end of the string as well: it wants to match the whole string!

You can see this difference in the following code:

>>> text = 'More with less'
>>> re.match('More', text)
<re.Match object; span=(0, 4), match='More'>
>>> re.fullmatch('More', text)
>>> 

The re.match(‘More’, text) method matches the string ‘More’ at the beginning of the string ‘More with less’. But the re.fullmatch(‘More’, text) method does not match the whole text. Therefore, it returns the None object—nothing is printed to your shell!

What’s the Difference Between re.fullmatch() and re.findall()?

There are two differences between the re.fullmatch(pattern, string) and re.findall(pattern, string) methods:

• re.fullmatch(pattern, string) returns a match object while re.findall(pattern, string) returns a list of matching strings.
• re.fullmatch(pattern, string) can only match the whole string, while re.findall(pattern, string) can return multiple matches in the string.

Both can be seen in the following example:

>>> text = 'the 42th truth is 42'
>>> re.fullmatch('.*?42', text)
<re.Match object; span=(0, 20), match='the 42th truth is 42'>
>>> re.findall('.*?42', text)
['the 42', 'th truth is 42']

Note that the regex .*? matches an arbitrary number of characters but it attempts to consume as few characters as possible. This is called “non-greedy” match (the *? operator). The fullmatch() method only returns a match object that matches the whole string. The findall() method returns a list of all occurrences. As the match is non-greedy, it finds two such matches.

What’s the Difference Between re.fullmatch() and re.search()?

The methods re.fullmatch() and re.search(pattern, string) both return a match object. However, re.fullmatch() attempts to match the whole string while re.search() matches anywhere in the string.

You can see this difference in the following code:

>>> text = 'Finxter is fun!'
>>> re.search('Finxter', text)
<re.Match object; span=(0, 7), match='Finxter'>
>>> re.fullmatch('Finxter', text)
>>> 

The re.search() method retrieves the match of the ‘Finxter’ substring as a match object. But the re.fullmatch() method has no return value because the substring ‘Finxter’ does not match the whole string ‘Finxter is fun!’.

How to Use the Optional Flag Argument?

As you’ve seen in the specification, the fullmatch() method comes with an optional third ‘flag’ argument:

re.fullmatch(pattern, string, flags=0)

What’s the purpose of the flags argument?

Flags allow you to control the regular expression engine. Because regular expressions are so powerful, they are a useful way of switching on and off certain features (for example, whether to ignore capitalization when matching your regex).

Syntax	Meaning
re.ASCII	If you don’t use this flag, the special Python regex symbols w, W, b, B, d, D, s and S will match Unicode characters. If you use this flag, those special symbols will match only ASCII characters — as the name suggests.
re.A	Same as re.ASCII
re.DEBUG	If you use this flag, Python will print some useful information to the shell that helps you debugging your regex.
re.IGNORECASE	If you use this flag, the regex engine will perform case-insensitive matching. So if you’re searching for [A-Z], it will also match [a-z].
re.I	Same as re.IGNORECASE
re.LOCALE	Don’t use this flag — ever. It’s depreciated—the idea was to perform case-insensitive matching depending on your current locale. But it isn’t reliable.
re.L	Same as re.LOCALE
re.MULTILINE	This flag switches on the following feature: the start-of-the-string regex ‘^’ matches at the beginning of each line (rather than only at the beginning of the string). The same holds for the end-of-the-string regex ‘$’ that now matches also at the end of each line in a multi-line string.
re.M	Same as re.MULTILINE
re.DOTALL	Without using this flag, the dot regex ‘.’ matches all characters except the newline character ‘n’. Switch on this flag to really match all characters including the newline character.
re.S	Same as re.DOTALL
re.VERBOSE	To improve the readability of complicated regular expressions, you may want to allow comments and (multi-line) formatting of the regex itself. This is possible with this flag: all whitespace characters and lines that start with the character ‘#’ are ignored in the regex.
re.X	Same as re.VERBOSE

Here’s how you’d use it in a practical example:

>>> text = 'Python is great!'
>>> re.search('PYTHON', text, flags=re.IGNORECASE)
<re.Match object; span=(0, 6), match='Python'>

Although your regex ‘PYTHON’ is all-caps, we ignore the capitalization by using the flag re.IGNORECASE.

Where to Go From Here?

This article has introduced the re.fullmatch(pattern, string) method that attempts to match the whole string—and returns a match object if it succeeds or None if it doesn’t.

Learning Python is hard. But if you cheat, it isn’t as hard as it has to be:

Download 8 Free Python Cheat Sheets now!

Why have regular expressions survived seven decades of technological disruption? Because coders who understand regular expressions have a massive advantage when working with textual data. They can write in a single line of code what takes others dozens!

This article is all about the match() method of Python’s re library. There are two similar methods to help you use regular expressions:

• The easy-to-use but less powerful findall() method returns a list of string matches. Check out our blog tutorial.
• The search() method returns a match object of the first match. Check out our blog tutorial.

So how does the re.match() method work? Let’s study the specification.

How Does re.match() Work in Python?

The re.match(pattern, string) method matches the pattern at the beginning of the string and returns a match object.

Specification:

re.match(pattern, string, flags=0)

The re.match() method has up to three arguments.

• pattern: the regular expression pattern that you want to match.
• string: the string which you want to search for the pattern.
• flags (optional argument): a more advanced modifier that allows you to customize the behavior of the function. Want to know how to use those flags? Check out this detailed article on the Finxter blog.

We’ll explore them in more detail later.

Return Value:

The re.match() method returns a match object. You may ask (and rightly so):

What’s a Match Object?

If a regular expression matches a part of your string, there’s a lot of useful information that comes with it: what’s the exact position of the match? Which regex groups were matched—and where?

The match object is a simple wrapper for this information. Some regex methods of the re package in Python—such as match()—automatically create a match object upon the first pattern match.

At this point, you don’t need to explore the match object in detail. Just know that we can access the start and end positions of the match in the string by calling the methods m.start() and m.end() on the match object m:

>>> m = re.match('h...o', 'hello world')
>>> m.start()
0
>>> m.end()
5
>>> 'hello world'[m.start():m.end()] 'hello'

In the first line, you create a match object m by using the re.match() method. The pattern ‘h…o’ matches in the string ‘hello world’ at start position 0. You use the start and end position to access the substring that matches the pattern (using the popular Python technique of slicing). But note that as the match() method always attempts to match only at the beginning of the string, the m.start() method will always return zero.

Now, you know the purpose of the match() object in Python. Let’s check out a few examples of re.match()!

A Guided Example for re.match()

First, you import the re module and create the text string to be searched for the regex patterns:

>>> import re
>>> text = ''' Ha! let me see her: out, alas! he's cold: Her blood is settled, and her joints are stiff; Life and these lips have long been separated: Death lies on her like an untimely frost Upon the sweetest flower of all the field. '''

Let’s say you want to search the text for the string ‘her’:

>>> re.match('lips', text)
>>>

The first argument is the pattern to be found: the string ‘lips’. The second argument is the text to be analyzed. You stored the multi-line string in the variable text—so you take this as the second argument. The third argument flags of the match() method is optional.

There’s no output! This means that the re.match() method did not return a match object. Why? Because at the beginning of the string, there’s no match for the regex pattern ‘lips’.

So how can we fix this? Simple, by matching all the characters that preced the string ‘lips’ in the text:

>>> re.match('(.|\n)*lips', text)
<re.Match object; span=(0, 122), match="\n Ha! let me see her: out, alas! he's cold:\n>

The regex (.|\n)*lips matches all prefixes (an arbitrary number of characters including new lines) followed by the string ‘lips’. This results in a new match object that matches a huge substring from position 0 to position 122. Note that the match object doesn’t print the whole substring to the shell. If you access the matched substring, you’ll get the following result:

>>> m = re.match('(.|\n)*lips', text)
>>> text[m.start():m.end()] "\n Ha! let me see her: out, alas! he's cold:\n Her blood is settled, and her joints are stiff;\n Life and these lips"

Interestingly, you can also achieve the same thing by specifying the third flag argument as follows:

>>> m = re.match('.*lips', text, flags=re.DOTALL)
>>> text[m.start():m.end()] "\n Ha! let me see her: out, alas! he's cold:\n Her blood is settled, and her joints are stiff;\n Life and these lips"

The re.DOTALL flag ensures that the dot operator . matches all characters including the new line character.

What’s the Difference Between re.match() and re.findall()?

There are two differences between the re.match(pattern, string) and re.findall(pattern, string) methods:

• re.match(pattern, string) returns a match object while re.findall(pattern, string) returns a list of matching strings.
• re.match(pattern, string) returns only the first match in the string—and only at the beginning—while re.findall(pattern, string) returns all matches in the string.

Both can be seen in the following example:

>>> text = 'Python is superior to Python'
>>> re.match('Py...n', text)
<re.Match object; span=(0, 6), match='Python'>
>>> re.findall('Py...n', text)
['Python', 'Python']

The string ‘Python is superior to Python’ contains two occurrences of ‘Python’. The match() method only returns a match object of the first occurrence. The findall() method returns a list of all occurrences.

What’s the Difference Between re.match() and re.search()?

The methods re.search(pattern, string) and re.match(pattern, string) both return a match object of the first match. However, re.match() attempts to match at the beginning of the string while re.search() matches anywhere in the string.

You can see this difference in the following code:

>>> text = 'Slim Shady is my name'
>>> re.search('Shady', text)
<re.Match object; span=(5, 10), match='Shady'>
>>> re.match('Shady', text)
>>>

The re.search() method retrieves the match of the ‘Shady’ substring as a match object. But if you use the re.match() method, there is no match and no return value because the substring ‘Shady’ does not occur at the beginning of the string ‘Slim Shady is my name’.

How to Use the Optional Flag Argument?

As you’ve seen in the specification, the match() method comes with an optional third ‘flag’ argument:

re.match(pattern, string, flags=0)

What’s the purpose of the flags argument?

Flags allow you to control the regular expression engine. Because regular expressions are so powerful, they are a useful way of switching on and off certain features (for example, whether to ignore capitalization when matching your regex).

Syntax	Meaning
re.ASCII	If you don’t use this flag, the special Python regex symbols w, W, b, B, d, D, s and S will match Unicode characters. If you use this flag, those special symbols will match only ASCII characters — as the name suggests.
re.A	Same as re.ASCII
re.DEBUG	If you use this flag, Python will print some useful information to the shell that helps you debugging your regex.
re.IGNORECASE	If you use this flag, the regex engine will perform case-insensitive matching. So if you’re searching for [A-Z], it will also match [a-z].
re.I	Same as re.IGNORECASE
re.LOCALE	Don’t use this flag — ever. It’s depreciated—the idea was to perform case-insensitive matching depending on your current locale. But it isn’t reliable.
re.L	Same as re.LOCALE
re.MULTILINE	This flag switches on the following feature: the start-of-the-string regex ‘^’ matches at the beginning of each line (rather than only at the beginning of the string). The same holds for the end-of-the-string regex ‘$’ that now matches also at the end of each line in a multi-line string.
re.M	Same as re.MULTILINE
re.DOTALL	Without using this flag, the dot regex ‘.’ matches all characters except the newline character ‘n’. Switch on this flag to really match all characters including the newline character.
re.S	Same as re.DOTALL
re.VERBOSE	To improve the readability of complicated regular expressions, you may want to allow comments and (multi-line) formatting of the regex itself. This is possible with this flag: all whitespace characters and lines that start with the character ‘#’ are ignored in the regex.
re.X	Same as re.VERBOSE

Here’s how you’d use it in a practical example:

>>> text = 'Python is great!'
>>> re.search('PYTHON', text, flags=re.IGNORECASE)
<re.Match object; span=(0, 6), match='Python'>

Although your regex ‘PYTHON’ is all-caps, we ignore the capitalization by using the flag re.IGNORECASE.

Where to Go From Here?

This article has introduced the re.match(pattern, string) method that attempts to match the first occurrence of the regex pattern at the beginning of a given string—and returns a match object.

Python soars in popularity. There are two types of people: those who understand coding and those who don’t. The latter will have larger and larger difficulties participating in the era of massive adoption and penetration of digital content. Do you want to increase your Python skills daily without investing a lot of time?

Then join my “Coffee Break Python” email list of tens of thousands of ambitious coders!

When I first learned about regular expressions, I didn’t appreciate their power. But there’s a reason regular expressions have survived seven decades of technological disruption: coders who understand regular expressions have a massive advantage when working with textual data. They can write in a single line of code what takes others dozens!

This article is all about the search() method of Python’s re library. To learn about the easy-to-use but less powerful findall() method that returns a list of string matches, check out our article about the similar findall() method.

So how does the re.search() method work? Let’s study the specification.

How Does re.search() Work in Python?

The re.search(pattern, string) method matches the first occurrence of the pattern in the string and returns a match object.

Specification:

re.search(pattern, string, flags=0)

The re.search() method has up to three arguments.

• pattern: the regular expression pattern that you want to match.
• string: the string which you want to search for the pattern.
• flags (optional argument): a more advanced modifier that allows you to customize the behavior of the function. Want to know how to use those flags? Check out this detailed article on the Finxter blog.

We’ll explore them in more detail later.

Return Value:

The re.search() method returns a match object. You may ask (and rightly so):

What’s a Match Object?

If a regular expression matches a part of your string, there’s a lot of useful information that comes with it: what’s the exact position of the match? Which regex groups were matched—and where?

The match object is a simple wrapper for this information. Some regex methods of the re package in Python—such as search()—automatically create a match object upon the first pattern match.

At this point, you don’t need to explore the match object in detail. Just know that we can access the start and end positions of the match in the string by calling the methods m.start() and m.end() on the match object m:

>>> m = re.search('h...o', 'hello world')
>>> m.start()
0
>>> m.end()
5
>>> 'hello world'[m.start():m.end()] 'hello'

In the first line, you create a match object m by using the re.search() method. The pattern ‘h…o’ matches in the string ‘hello world’ at start position 0. You use the start and end position to access the substring that matches the pattern (using the popular Python technique of slicing).

Now, you know the purpose of the match() object in Python. Let’s check out a few examples of re.search()!

A Guided Example for re.search()

First, you import the re module and create the text string to be searched for the regex patterns:

>>> import re
>>> text = ''' Ha! let me see her: out, alas! he's cold: Her blood is settled, and her joints are stiff; Life and these lips have long been separated: Death lies on her like an untimely frost Upon the sweetest flower of all the field. '''

Let’s say you want to search the text for the string ‘her’:

>>> re.search('her', text)
<re.Match object; span=(20, 23), match='her'>

The first argument is the pattern to be found. In our case, it’s the string ‘her’. The second argument is the text to be analyzed. You stored the multi-line string in the variable text—so you take this as the second argument. You don’t need to define the optional third argument flags of the search() method because you’re fine with the default behavior in this case.

Look at the output: it’s a match object! The match object gives the span of the match—that is the start and stop indices of the match. We can also directly access those boundaries by using the start() and stop() methods of the match object:

>>> m = re.search('her', text)
>>> m.start()
20
>>> m.end()
23

The problem is that the search() method only retrieves the first occurrence of the pattern in the string. If you want to find all matches in the string, you may want to use the findall() method of the re library.

What’s the Difference Between re.search() and re.findall()?

There are two differences between the re.search(pattern, string) and re.findall(pattern, string) methods:

• re.search(pattern, string) returns a match object while re.findall(pattern, string) returns a list of matching strings.
• re.search(pattern, string) returns only the first match in the string while re.findall(pattern, string) returns all matches in the string.

Both can be seen in the following example:

>>> text = 'Python is superior to Python'
>>> re.search('Py...n', text)
<re.Match object; span=(0, 6), match='Python'>
>>> re.findall('Py...n', text)
['Python', 'Python']

The string ‘Python is superior to Python’ contains two occurrences of ‘Python’. The search() method only returns a match object of the first occurrence. The findall() method returns a list of all occurrences.

What’s the Difference Between re.search() and re.match()?

The methods re.search(pattern, string) and re.match(pattern, string) both return a match object of the first match. However, re.match() attempts to match at the beginning of the string while re.search() matches anywhere in the string.

You can see this difference in the following code:

>>> text = 'Slim Shady is my name'
>>> re.search('Shady', text)
<re.Match object; span=(5, 10), match='Shady'>
>>> re.match('Shady', text)
>>>

The re.search() method retrieves the match of the ‘Shady’ substring as a match object. But if you use the re.match() method, there is no match and no return value because the substring ‘Shady’ does not occur at the beginning of the string ‘Slim Shady is my name’.

How to Use the Optional Flag Argument?

As you’ve seen in the specification, the search() method comes with an optional third ‘flag’ argument:

re.search(pattern, string, flags=0)

What’s the purpose of the flags argument?

Flags allow you to control the regular expression engine. Because regular expressions are so powerful, they are a useful way of switching on and off certain features (for example, whether to ignore capitalization when matching your regex).

Syntax	Meaning
re.ASCII	If you don’t use this flag, the special Python regex symbols \w, \W, \b, \B, \d, \D, \s and \S will match Unicode characters. If you use this flag, those special symbols will match only ASCII characters — as the name suggests.
re.A	Same as re.ASCII
re.DEBUG	If you use this flag, Python will print some useful information to the shell that helps you debugging your regex.
re.IGNORECASE	If you use this flag, the regex engine will perform case-insensitive matching. So if you’re searching for [A-Z], it will also match [a-z].
re.I	Same as re.IGNORECASE
re.LOCALE	Don’t use this flag — ever. It’s depreciated—the idea was to perform case-insensitive matching depending on your current locale. But it isn’t reliable.
re.L	Same as re.LOCALE
re.MULTILINE	This flag switches on the following feature: the start-of-the-string regex ‘^’ matches at the beginning of each line (rather than only at the beginning of the string). The same holds for the end-of-the-string regex ‘$’ that now matches also at the end of each line in a multi-line string.
re.M	Same as re.MULTILINE
re.DOTALL	Without using this flag, the dot regex ‘.’ matches all characters except the newline character ‘\n’. Switch on this flag to really match all characters including the newline character.
re.S	Same as re.DOTALL
re.VERBOSE	To improve the readability of complicated regular expressions, you may want to allow comments and (multi-line) formatting of the regex itself. This is possible with this flag: all whitespace characters and lines that start with the character ‘#’ are ignored in the regex.
re.X	Same as re.VERBOSE

Here’s how you’d use it in a practical example:

>>> text = 'Python is great!'
>>> re.search('PYTHON', text, flags=re.IGNORECASE)
<re.Match object; span=(0, 6), match='Python'>

Although your regex ‘PYTHON’ is all-caps, we ignore the capitalization by using the flag re.IGNORECASE.

Where to Go From Here?

This article has introduced the re.search(pattern, string) method that attempts to match the first occurrence of the regex pattern in a given string—and returns a match object.

Python soars in popularity. There are two types of people: those who understand coding and those who don’t. The latter will have larger and larger difficulties participating in the era of massive adoption and penetration of digital content. Do you want to increase your Python skills daily without investing a lot of time?

Then join my “Coffee Break Python” email list of tens of thousands of ambitious coders!

In many functions, you see a third argument flags. What are they and how do they work?

Flags allow you to control the regular expression engine. Because regular expressions are so powerful, they are a useful way of switching on and off certain features (e.g. whether to ignore capitalization when matching your regex).

For example, here’s how the third argument flags is used in the re.findall() method:

re.findall(pattern, string, flags=0)

So the flags argument seems to be an integer argument with the default value of 0. To control the default regex behavior, you simply use one of the predefined integer values. You can access these predefined values via the re library:

Syntax	Meaning
re.ASCII	If you don’t use this flag, the special Python regex symbols \w, \W, \b, \B, \d, \D, \s and \S will match Unicode characters. If you use this flag, those special symbols will match only ASCII characters — as the name suggests.
re.A	Same as re.ASCII
re.DEBUG	If you use this flag, Python will print some useful information to the shell that helps you debugging your regex.
re.IGNORECASE	If you use this flag, the regex engine will perform case-insensitive matching. So if you’re searching for [A-Z], it will also match [a-z].
re.I	Same as re.IGNORECASE
re.LOCALE	Don’t use this flag — ever. It’s depreciated—the idea was to perform case-insensitive matching depending on your current locale. But it isn’t reliable.
re.L	Same as re.LOCALE
re.MULTILINE	This flag switches on the following feature: the start-of-the-string regex ‘^’ matches at the beginning of each line (rather than only at the beginning of the string). The same holds for the end-of-the-string regex ‘$’ that now matches also at the end of each line in a multi-line string.
re.M	Same as re.MULTILINE
re.DOTALL	Without using this flag, the dot regex ‘.’ matches all characters except the newline character ‘\n’. Switch on this flag to really match all characters including the newline character.
re.S	Same as re.DOTALL
re.VERBOSE	To improve the readability of complicated regular expressions, you may want to allow comments and (multi-line) formatting of the regex itself. This is possible with this flag: all whitespace characters and lines that start with the character ‘#’ are ignored in the regex.
re.X	Same as re.VERBOSE

How to Use These Flags?

Simply include the flag as the optional flag argument as follows:

import re text = ''' Ha! let me see her: out, alas! he's cold: Her blood is settled, and her joints are stiff; Life and these lips have long been separated: Death lies on her like an untimely frost Upon the sweetest flower of all the field. ''' print(re.findall('HER', text, flags=re.IGNORECASE))
# ['her', 'Her', 'her', 'her']

As you see, the re.IGNORECASE flag ensures that all occurrences of the string ‘her’ are matched—no matter their capitalization.

How to Use Multiple Flags?

Yes, simply add them together (sum them up) as follows:

import re text = ''' Ha! let me see her: out, alas! he's cold: Her blood is settled, and her joints are stiff; Life and these lips have long been separated: Death lies on her like an untimely frost Upon the sweetest flower of all the field. ''' print(re.findall(' HER # Ignored', text, flags=re.IGNORECASE + re.VERBOSE))
# ['her', 'Her', 'her', 'her']

You use both flags re.IGNORECASE (all occurrences of lower- or uppercase string variants of ‘her’ are matched) and re.VERBOSE (ignore comments and whitespaces in the regex). You sum them together re.IGNORECASE + re.VERBOSE to indicate that you want to take both.

When I first learned about regular expressions, I didn’t really appreciate their power. But there’s a reason regular expressions have survived seven decades of technological disruption: coders who understand regular expressions have a massive advantage when working with textual data. They can write in a single line of code what takes others dozens!

This article is all about the findall() method of Python’s re library. The findall() method is the most basic way of using regular expressions in Python: If you want to master them, start here!

So how does the re.findall() method work? Let’s study the specification.

How Does the findall() Method Work in Python?

The re.findall(pattern, string) method finds all occurrences of the pattern in the string and returns a list of all matching substrings.

Specification:

re.findall(pattern, string, flags=0)

The re.findall() method has up to three arguments.

• pattern: the regular expression pattern that you want to match.
• string: the string which you want to search for the pattern.
• flags (optional argument): a more advanced modifier that allows you to customize the behavior of the function. Want to know how to use those flags? Check out this detailed article on the Finxter blog.

We will have a look at each of them in more detail.

Return Value:

The re.findall() method returns a list of strings. Each string element is a matching substring of the string argument.

Let’s check out a few examples!

Examples re.findall()

First, you import the re module and create the text string to be searched for the regex patterns:

import re text = ''' Ha! let me see her: out, alas! he's cold: Her blood is settled, and her joints are stiff; Life and these lips have long been separated: Death lies on her like an untimely frost Upon the sweetest flower of all the field. '''

Let’s say, you want to search the text for the string ‘her’:

>>> re.findall('her', text)
['her', 'her', 'her']

The first argument is the pattern you look for. In our case, it’s the string ‘her’. The second argument is the text to be analyzed. You stored the multi-line string in the variable text—so you take this as the second argument. You don’t need to define the optional third argument flags of the findall() method because you’re fine with the default behavior in this case.

Also note that the findall() function returns a list of all matching substrings. In this case, this may not be too useful because we only searched for an exact string. But if we search for more complicated patterns, this may actually be very useful:

>>> re.findall('\\bf\w+\\b', text)
['frost', 'flower', 'field']

The regex ‘\\bf\w+\\b’ matches all words that start with the character ‘f’.

You may ask: why to enclose the regex with a leading and trailing ‘\\b’? This is the word boundary character that matches the empty string at the beginning or at the end of a word. You can define a word as a sequence of characters that are not whitespace characters or other delimiters such as ‘.:,?!’.

In the previous example, you need to escape the boundary character ‘\b’ again because in a Python string, the default meaning of the character sequence ‘\b’ is the backslash character.

Where to Go From Here?

This article has introduced the re.findall(pattern, string) method that attempts to match all occurrences of the regex pattern in a given string—and returns a list of all matches as strings.

Python is growing rapidly and the world is more and more divided into two classes: those who understand coding and those who don’t. The latter will have larger and larger difficulties participating in the era of massive adoption and penetration of digital content. Do you want to increase your Python skills on a daily basis without investing a lot of time?

Then join my “Coffee Break Python” email list of tens of thousands of ambitious coders!