<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<!--Converted with LaTeX2HTML 98.2 beta6 (August 14th, 1998)
original version by:  Nikos Drakos, CBLU, University of Leeds
* revised and updated by:  Marcus Hennecke, Ross Moore, Herb Swan
* with significant contributions from:
  Jens Lippmann, Marek Rouchal, Martin Wilck and others -->
<HTML>
<HEAD>
<TITLE>2. Lexical analysis</TITLE>
<META NAME="description" CONTENT="2. Lexical analysis">
<META NAME="keywords" CONTENT="ref">
<META NAME="resource-type" CONTENT="document">
<META NAME="distribution" CONTENT="global">
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<LINK REL="STYLESHEET" HREF="ref.css">
<LINK REL="next" HREF="datamodel.html">
<LINK REL="previous" HREF="introduction.html">
<LINK REL="up" HREF="ref.html">
<LINK REL="next" HREF="line-structure.html">
</HEAD>
<BODY bgcolor="#ffffff">
<div class=navigation>
<table align=center width="100%" cellpadding=0 cellspacing=2>
<tr>
<td><A HREF="line-structure.html"><img width=32 height=32 align=bottom border=0 alt="next"
 src="../icons/next.gif"></A></td>
<td><A HREF="ref.html"><img width=32 height=32 align=bottom border=0 alt="up"
 src="../icons/up.gif"></A></td>
<td><A HREF="notation.html"><img width=32 height=32 align=bottom border=0 alt="previous"
 src="../icons/previous.gif"></A></td>
<td align=center bgcolor="#99CCFF" width="100%">
 <b class=title>Python Reference Manual</b></td>
<td><A HREF="contents.html"><img width=32 height=32 align=bottom border=0 alt="contents"
 src="../icons/contents.gif"></A></td>
<td><img width=32 height=32 align=bottom border=0 alt=""
 src="../icons/blank.gif"></td>
<td><A HREF="genindex.html"><img width=32 height=32 align=bottom border=0 alt="index"
 src="../icons/index.gif"></A></td>
</tr></table><b class=navlabel>Next:</b> <span class=sectref><A HREF="line-structure.html">2.1 Line structure</A></span>
<b class=navlabel>Up:</b> <span class=sectref><A HREF="ref.html">Python Reference Manual</A></span>
<b class=navlabel>Previous:</b> <span class=sectref><A HREF="notation.html">1.1 Notation</A></span>
<br><hr></div>
<!--End of Navigation Panel-->

<H1><A NAME="SECTION004000000000000000000"></A>
<BR>
2. Lexical analysis
</H1>

<P>
A Python program is read by a <i>parser</i>.  Input to the parser is a
stream of <i>tokens</i>, generated by the <i>lexical analyzer</i>.  This
chapter describes how the lexical analyzer breaks a file into tokens.
<a name="l2h-7"></a>
<a name="l2h-8"></a>
<a name="l2h-9"></a>

<P>
Python uses the 7-bit ASCII character set for program text and string
literals. 8-bit characters may be used in string literals and comments
but their interpretation is platform dependent; the proper way to
insert 8-bit characters in string literals is by using octal or
hexadecimal escape sequences.

<P>
The run-time character set depends on the I/O devices connected to the
program but is generally a superset of ASCII.

<P>
<b>Future compatibility note:</b> It may be tempting to assume that the
character set for 8-bit characters is ISO Latin-1 (an ASCII
superset that covers most western languages that use the Latin
alphabet), but it is possible that in the future Unicode text editors
will become common.  These generally use the UTF-8 encoding, which is
also an ASCII superset, but with very different use for the
characters with ordinals 128-255.  While there is no consensus on this
subject yet, it is unwise to assume either Latin-1 or UTF-8, even
though the current implementation appears to favor Latin-1.  This
applies both to the source character set and the run-time character
set.

<P>

<p><hr>
<!--Table of Child-Links-->


<UL>
<LI><A NAME="tex2html219"
 HREF="line-structure.html">2.1 Line structure</A>
<UL>
<LI><A NAME="tex2html220"
 HREF="logical.html">2.1.1 Logical lines</A>
<LI><A NAME="tex2html221"
 HREF="physical.html">2.1.2 Physical lines</A>
<LI><A NAME="tex2html222"
 HREF="comments.html">2.1.3 Comments</A>
<LI><A NAME="tex2html223"
 HREF="explicit-joining.html">2.1.4 Explicit line joining</A>
<LI><A NAME="tex2html224"
 HREF="implicit-joining.html">2.1.5 Implicit line joining</A>
<LI><A NAME="tex2html225"
 HREF="blank-lines.html">2.1.6 Blank lines <a name="l2h-21"></a></A>
<LI><A NAME="tex2html226"
 HREF="indentation.html">2.1.7 Indentation</A>
<LI><A NAME="tex2html227"
 HREF="whitespace.html">2.1.8 Whitespace between tokens</A>
</UL>
<LI><A NAME="tex2html228"
 HREF="other-tokens.html">2.2 Other tokens</A>
<LI><A NAME="tex2html229"
 HREF="identifiers.html">2.3 Identifiers and keywords</A>
<UL>
<LI><A NAME="tex2html230"
 HREF="keywords.html">2.3.1 Keywords</A>
<LI><A NAME="tex2html231"
 HREF="id-classes.html">2.3.2 Reserved classes of identifiers</A>
</UL>
<LI><A NAME="tex2html232"
 HREF="literals.html">2.4 Literals</A>
<UL>
<LI><A NAME="tex2html233"
 HREF="strings.html">2.4.1 String literals</A>
<LI><A NAME="tex2html234"
 HREF="string-catenation.html">2.4.2 String literal concatenation</A>
<LI><A NAME="tex2html235"
 HREF="numbers.html">2.4.3 Numeric literals</A>
<LI><A NAME="tex2html236"
 HREF="integers.html">2.4.4 Integer and long integer literals</A>
<LI><A NAME="tex2html237"
 HREF="floating.html">2.4.5 Floating point literals</A>
<LI><A NAME="tex2html238"
 HREF="imaginary.html">2.4.6 Imaginary literals</A>
</UL>
<LI><A NAME="tex2html239"
 HREF="operators.html">2.5 Operators</A>
<LI><A NAME="tex2html240"
 HREF="delimiters.html">2.6 Delimiters</A>
</UL>
<!--End of Table of Child-Links-->
<p>
<div class=navigation><hr><table align=center width="100%" cellpadding=0 cellspacing=2>
<tr>
<td><A HREF="line-structure.html"><img width=32 height=32 align=bottom border=0 alt="next"
 src="../icons/next.gif"></A></td>
<td><A HREF="ref.html"><img width=32 height=32 align=bottom border=0 alt="up"
 src="../icons/up.gif"></A></td>
<td><A HREF="notation.html"><img width=32 height=32 align=bottom border=0 alt="previous"
 src="../icons/previous.gif"></A></td>
<td align=center bgcolor="#99CCFF" width="100%">
 <b class=title>Python Reference Manual</b></td>
<td><A HREF="contents.html"><img width=32 height=32 align=bottom border=0 alt="contents"
 src="../icons/contents.gif"></A></td>
<td><img width=32 height=32 align=bottom border=0 alt=""
 src="../icons/blank.gif"></td>
<td><A HREF="genindex.html"><img width=32 height=32 align=bottom border=0 alt="index"
 src="../icons/index.gif"></A></td>
</tr></table><b class=navlabel>Next:</b> <span class=sectref><A HREF="line-structure.html">2.1 Line structure</A></span>
<b class=navlabel>Up:</b> <span class=sectref><A HREF="ref.html">Python Reference Manual</A></span>
<b class=navlabel>Previous:</b> <span class=sectref><A HREF="notation.html">1.1 Notation</A></span>
</div>
<!--End of Navigation Panel-->
<ADDRESS>
<hr>Send comments on this document to <a href="mailto:python-docs@python.org">python-docs@python.org</a>.
</ADDRESS>
</BODY>
</HTML>
