<?xml version="1.0"?>
<!DOCTYPE webpage PUBLIC "-//Norman Walsh//DTD Website V2.1b1//EN" "../../../dtd/website.dtd" [
  <!ENTITY applications SYSTEM "applications.xml" NDATA XML>
  <!ENTITY bases SYSTEM "bases.xml" NDATA XML>
  <!ENTITY entrepots SYSTEM "entrepots.xml" NDATA XML>
  <!ENTITY params SYSTEM "../reference/parametres_sdx.xml" NDATA XML>
  <!ENTITY xconf SYSTEM "../configuration/configuration.xml" NDATA XML>
]>

<webpage id="multilinguisme"> 

  <head>
	 <title>SDX and multilingualism</title>
	 <titleabbrev>Multilingualism</titleabbrev>
	 <author>
		<firstname>Martin</firstname>
		<surname>Sévigny</surname>
		<affiliation>
		  <orgname>AJLSM</orgname> 
		</affiliation>
	 </author>
	 <revhistory>
		<revision>
		  <revnumber>1.0</revnumber> 
		  <date>2003-06-03</date> 
		</revision>
	 </revhistory>
  </head>

  <para>SDX is a tool for searching and displaying XML documents, which emphasizes <emphasis>language</emphasis> issues. That is the reason why it is presented as <emphasis>multilingual</emphasis>. This concept is quite large and we will describe, on the one hand what it means for SDX, on the other hand which are the <emphasis>multilingual services</emphasis> offered by SDX. Avant d'y arriver, il est important de rappeler un fait important à ce sujet:</para>
  <blockquote> 
	 <para><emphasis role="bold">SDX est une plate-forme qui permet de réaliser des applications de recherche multilingues</emphasis>; this does not mean that SDX already includes all necessary tools for processing correctly all languages which a developer may wish to implement.</para>
  </blockquote> 
  <section> 
	 <title>What is a language for SDX ?</title>
	 <para>All discussion on multilingualism must, beforehands, define what a language is in SDX-2. Especially because SDX has been built upon two technologies which include their own conception of a language (Java et XML); good news, both conceptions are not in opposition.</para>
	 <para>For Java, a language is actually a "local" (ie. class 
		<ulink url="http://java.sun.com/j2se/1.3/docs/api/java/util/Locale.html">java.util.Locale</ulink> for a full definition), which combines a language code (ISO-639), a country code (ISO-3166) and a "variant", optional, which allows to define several language options.</para>
	 <para>For XML, 
		<ulink url="http://www.w3.org/TR/REC-xml#sec-lang-tag">section 2.12</ulink> of the standard states what a language is and how it can be represented. XML includes a specific attribute (<sgmltag class="attribute">xml:lang</sgmltag>) to identify a language, this attribute must have a value which is defined in the 
		<ulink url="http://www.ietf.org/rfc/rfc1766.txt">IETF RFC-1766</ulink>. This standard defines that a language is specified by a combination of a language code, a country code according similar standards as Java, although this combination can be replaced by a 
		<ulink url="http://www.isi.edu/in-notes/iana/assignments/languages/">language code registrated in the IANA</ulink>.</para>
	 <para>From both definitions, it has been decided that, for SDX-2, a language is a combination of an ISO-639 language code, an ISO-3166 country code (optional) and a variant (optional). However, for representing those languages in an XML environment, the xml:lang attribute is used for the two first parts, and a "variant" attrubute is used for the third one. That is the reason why it is possible to set, for example:</para>
	 <programlisting>
  &lt;sdx:field xml:lang="fr-CA" variant="qc".../&gt;

</programlisting> 
	 <para>However, for the language of interface, which is dealt with in the following section, the concept of variant is never used; only a language code and a country code are used, both are identified with a <sgmltag class="attribute">xml:lang</sgmltag> attribute for example.</para>
  </section> 
  <section> 
	 <title>Language of interfaces</title>
	 <para>Web applications which are built with SDX usually make dynamic Web pages with the XSP language, and XSP usually includes an <sgmltag class="element">sdx:page</sgmltag> element. That element then allows to use specific SDX services. This element, whatever it contains, has a great impact: the XML document (virtual) generated by the XSP page will have <sgmltag class="element">sdx:document</sgmltag> as root element, and this element will contain <sgmltag class="attribute">xml:lang</sgmltag> as an attribute which identifies the language desired for the interface. For example, we may find:</para>
	 <programlisting>
&lt;sdx:document xml:lang="fr-CA"&gt;
  ...
  &lt;sdx:results ...&gt;
    ...
  &lt;/sdx:results&gt;
&lt;/sdx:document&gt;
</programlisting> 
	 <para>This attribute always exists, it must be used as a basis for creating multilingual interfaces. In an XSLT transformation applied to this virtual document to make an HTML page (this is only an example), this information can be used to generate a different content according to the language:</para>
	 <programlisting>
...
&lt;xsl:variable name="l" select="string(/sdx:document/@xml:lang)"/&gt;
...
&lt;xsl:choose&gt;
  &lt;xsl:when test="starts-with($l, 'fr')"&gt;Bonjour&lt;/xsl:when&gt;
  &lt;xsl:when test="starts-with($l, 'en')"&gt;Hello&lt;/xsl:when&gt;
  ...
&lt;/xsl:choose&gt;
...
</programlisting> 
	 <para>This feature appears interesting (though it has nothing revolutionary), but it raises a key issue: <emphasis role="bold">how does SDX determine the language which is specified in this attribute </emphasis> ? Actually, SDX will search this information in various locations, in a specific order, and the first language it identifies is the one it will write in the attribute. We will document those locations where the language is defined and in which order they are checked.</para>
	 <section> 
		<title>where is the language ?</title>
		<section> 
		  <title>The language of the application</title>
		  <para>In the 
			 <olink targetdocent="xconf">configuration file</olink> (<filename>application.xconf</filename>), The superior element <sgmltag class="element">sdx:application</sgmltag> must have an attribute<sgmltag class="attribute">xml:lang</sgmltag>. The value of this attribute can be used by SDX to identify the display language. This value can be considered as the default language of the application; for a single language application this attribute is usually enough to indicate the language.</para>
		</section> 
		<section> 
		  <title>The user language</title>
		  <para>Web applications which are built with SDX contain user services for providing Web pages. Even when there is no formal identification, SDX consider a specific user, the <emphasis>anonymous</emphasis>user; no language is linked to that user, but it is an exception. All other users have an associated language, and this language can be used to determine the display language.</para>
		  <para>In the SDX interface for user creation, a language can be specified for a user. Applications with their own interface of user management can link a language and a method, SDX will still use that language to specify the display language.</para>
		</section> 
		<section> 
		  <title>The language of the XSP page</title>
		  <para>In an XSP page, a language can be directly specified and it is possible for SDX to use that language for the display language. This language specification must be done with an <sgmltag class="attribute">xml:lang</sgmltag> attribute of the <sgmltag class="element">sdx:page</sgmltag> element or its parent element <sgmltag class="element">xsp:page</sgmltag>. For example:</para>
		  <programlisting>
&lt;xsp:page&gt;
  &lt;sdx:page xml:lang="es"&gt;
  ...
  &lt;/sdx:page&gt;
&lt;/xsp:page&gt;
</programlisting> 
		</section> 
		<section> 
		  <title>Dynamic language parameter</title>
		  <para>The SDX API uses a generic concept of parameters and proposes various ways of specifying those parameters in an XSP page. Further details can be found in the documentation on the concept of parameters, but we will illustrate their use for the language.</para>
		  <para>SDX parameters must have a name; for the language, the parameter name is <emphasis role="code">lang</emphasis>. Then, SDX offers five ways to specify a value of a parameter, which are, ordered by priority: a Java variable which is not null and is definied in the XSP; a value which comes from an HTTP request parameter (thus from the URL); a value which is stocked in a session object on the server; an explicit value in the XSP page, finally a default value in SDX. We must also mention that for the language, there is no default value in SDX; this feature is not important, since the application language (defined in the <filename>application.xconf</filename>) is considered as the default value. We shall also mention that using an explicit value for a parameter does not have any advantage compared with using the <sgmltag class="attribute">xml:lang</sgmltag> attribute in the XSP
			 page, as mentioned above.</para>
		  <para>This parameter <emphasis role="code">code</emphasis> must be associated to the <sgmltag class="element">sdx:page</sgmltag> element. The following example shows various ways to do this:</para>
		  <programlisting>
&lt;!-- Valeur explicite --&gt;
&lt;sdx:page lang="fr"&gt;

&lt;!-- Value which comes from the session object whose key is "language" --&gt;
&lt;sdx:page langSession="language"&gt;

&lt;!-- Value which come from an HTTP parameter "l" (for example the test.xsp?l=fr URL) --&gt;
&lt;sdx:page langParam="l"&gt;

&lt;!-- Value which comes from a Java variable which has been previously defined --&gt;
&lt;xsp:logic&gt;
  String l = "fr";
&lt;/xsp:logic&gt;
&lt;sdx:page langString="l"&gt;
</programlisting> 
		  <para>Thos examples all use the attributes method to identify a parameter. But SDX offers a second method, which is sub-elements <sgmltag class="element">sdx:parameter</sgmltag> to specify parameters. It is therefore possible to replace the third example above by the following code:</para>
		  <programlisting>
&lt;sdx:page&gt;
  &lt;sdx:parameter name="lang" valueParam="l"/&gt;
</programlisting> 
		  <para>This method of parameters is very flexible, and it is necessary to create genuine dynamic and multilingual Web applications. For example, it is possible to add the <emphasis role="code">l</emphasis> parameter to all interfaces and to give them the language value, and to use the <sgmltag class="attribute">langParam="l"</sgmltag> attribute in the XSP page, so that the display language always has a suitable value. </para>
		  <para>However, this approach has a drawback: the developer must always include the <emphasis role="code">l</emphasis> parameter in the hypertextual links, which may lead to mistakes. As a solution, it is possible to stock that value in an object session, SDX can then use it. It is only necessary to use this in the XSP pages:</para>
		  <programlisting>
&lt;sdx:page langSession="l" langParam="l"&gt;</programlisting> 
		  <para>Specifying both indications is not a problem, on the contrary, since SDX defines a priority: the URL parameter (<sgmltag class="attribute">langParam</sgmltag> attribute is a priority on the session object, but hat priority is only considered if there is indeed that parameter in the URL. Moreover, if the URL contains that parameter, then SDX uses it and keeps its value as a session object, because of the <sgmltag class="attribute">langSession</sgmltag> attribute with the same value as the one of the <sgmltag class="attribute">langParam</sgmltag> attribute.</para>
		  <para>This feature of SDX parameters is probably the one which leads to the best flexibility and ease of use for language management. This can be illustrated with an example. Assuming that the above code belongs to an XSP page called <filename>index.xsp</filename>. This page is successively called with various parameter values, and let's consider the way the display language is specified:</para>
		  <orderedlist>
			 <listitem>
				<para><filename>index.xsp</filename>: the language is defined in the session object of which the key is <emphasis role="code">code</emphasis>. If it does not exist, then SDX looks for the language, somewhere else.</para>
			 </listitem><listitem>
				<para><filename>index.xsp?l=en</filename>: the language will be <emphasis role="code">en</emphasis>.</para>
			 </listitem><listitem>
				<para><filename>index.xsp</filename>: language does not change, it is still <emphasis role="code">en</emphasis>.</para>
			 </listitem><listitem>
				<para><filename>index.xsp?l=fr</filename>: the language changes for <emphasis role="code">fr</emphasis>.</para>
			 </listitem>
		  </orderedlist>
		  <para>This approach allows the developer to specify a URL parameter only when he wants to change the display language. It is also possible to add a <sgmltag class="attribute">langString</sgmltag> attribute which behaves the same way as <sgmltag class="attribute">langParam</sgmltag>, but with a value defined by a Java variable and with an even higher priority. This allows to specify the language according to a logic which is more precise than the one entailed by simple URL parameter values, if necessary.</para>
		</section> 
	 </section> 
	 <section> 
		<title>Search order</title>
		<para>Sdx can search language information in dynamic parameters, explicitely in the XSP page, in the current user information and in the configuration of the application. All these locations can contain language information, tehrefore a priority must be defined. Dynamic information should logically be the priority; We also apply a second rule which is that information which is contained in the XSP page is prioritary.</para>
		<para>Consequently, The priority order is:</para>
		<orderedlist>
		  <listitem>
			 <para>Dynamic parameters</para>
		  </listitem><listitem>
			 <para>Language of the XSP page</para>
		  </listitem><listitem>
			 <para>The user language</para>
		  </listitem><listitem>
			 <para>The application language (<filename>application.xconf</filename>)</para>
		  </listitem>
		</orderedlist>
	 </section> 
  </section> 
  <section> 
	 <title>Language and search functionalities</title>
	 <para>The other important feature of multilingualism for SDX is the relationship with the search function. Indeed, every application which processes text is biased by the language of the texts it deals with. In documentary search, this bias appears in three different ways:</para>
	 <orderedlist>
		<listitem>
		  <para>When indexing and searching it, the text must be separated into <emphasis>words</emphasis>, this division depends on the language of text.</para>
		</listitem><listitem>
		  <para>When indexing and searching, it may be necessary to transform the text for, for example, putting it in minuscule characters, delete diacritics, etc. Once again, the text language biases this transformation.</para>
		</listitem><listitem>
		  <para>When ordering search results, according to textual inforation which are contained in various fields values. Ordering on textual content also depends on text language.</para>
		</listitem>
	 </orderedlist>
	 <para>SDX allows the applications developer to control the operations which depend on the language very precisely; this control mainly applies at the level of fields of the document base. When defining those fields, it is possible to set a language for them and, optionally, a specific word analyser. It is important to that the discussion below is specific to the use of the Lucene engine by SDX; possibly, if SDX allows to use additional search engines, those information may not apply.</para>
	 <section> 
		<title>The word analyser</title>
		<para>When SDX indexes textual content in field of type <emphasis role="code">word</emphasis>, textual content is first transformed by a <emphasis>word analyser</emphasis>. Indeed, each single word is stocked in the index, and those words can be transformed in order, for example, to get the whole texty in minuscule characters and delete diacritics. For example, when indexing textual content <emphasis>Première personne de la soirée</emphasis> (First person in the party), an analyser could provide three words to index: <emphasis>première</emphasis>, <emphasis>personne</emphasis> and <emphasis>soirée</emphasis>. The analyser is also in charge of deleting stop words. To make the system work, the same transformation must be applied, with the same word analyser, when executing a search request. If a user searches <emphasis>Soirée</emphasis>, the word analyser provides the word
		  <emphasis>soiree</emphasis>, which allows to find the document which contains the content which has been mentioned as an example.</para>
		<para>It would be difficult to consider a word analyser which would work well for all languages, since, word identification and transformation uselly depends on the language of text. That is the reason why SDX allows the application developer to use different word analysers for each field of the database. Those analysers can either be included in SDX, or provided by the developer himself; a word analyser is only a Java class which extends (directly or indirectly) the 
		  <ulink url="../../apidocs/fr/gouv/culture/sdx/search/lucene/analysis/Analyzer.html"><filename>fr.gouv.culture.sdx.search.lucene.analysis.Analyzer</filename></ulink> class. The following word analysers are included in SDX 2:</para>
		<itemizedlist>
		  <listitem>
			 <para><ulink url="../../apidocs/fr/gouv/culture/sdx/search/lucene/analysis/Analyzer_cn.html"><filename>fr.gouv.culture.sdx.search.lucene.analysis.Analyzer_cn</filename></ulink>&nbsp;: Chinese.</para>
		  </listitem><listitem>
			 <para><ulink url="../../apidocs/fr/gouv/culture/sdx/search/lucene/analysis/Analyzer_cn.html"><filename>fr.gouv.culture.sdx.search.lucene.analysis.Analyzer_de</filename></ulink>&nbsp;: German.</para>
		  </listitem><listitem>
			 <para><ulink url="../../apidocs/fr/gouv/culture/sdx/search/lucene/analysis/Analyzer_cn.html"><filename>fr.gouv.culture.sdx.search.lucene.analysis.Analyzer_en</filename></ulink>&nbsp;: English.</para>
		  </listitem><listitem>
			 <para><ulink url="../../apidocs/fr/gouv/culture/sdx/search/lucene/analysis/Analyzer_cn.html"><filename>fr.gouv.culture.sdx.search.lucene.analysis.Analyzer_fr</filename></ulink>&nbsp;: French.</para>
		  </listitem><listitem>
			 <para><ulink url="../../apidocs/fr/gouv/culture/sdx/search/lucene/analysis/Analyzer_cn.html"><filename>fr.gouv.culture.sdx.search.lucene.analysis.Analyzer_ru</filename></ulink>&nbsp;: Russian.</para>
		  </listitem>
		</itemizedlist>
		<para>Additional analysers may be included in SDX; every contributions will be welcomed.</para>
		<para>In order to make more flexible to use word analysers, or to allow analysers to be used in a slightly different manner, SDX includes a mechanism of configuration of analysers. This configuration is performed through an XML document which contains various information types, which can be specific to the analyser or apply to all analysers, unless there is an exception. A configuration file for each language is included in SDX, in a file which is located in the <filename>sdx/resources/conf/analysis</filename> from the SDX installation directory (for example <filename>.../webapps/sdx</filename>). Here is a part of the configuration file for French language:</para>
		<programlisting>
&lt;?xml version="1.0" encoding="ISO-8859-1"?&gt;
&lt;french useStopWords="true" keepAccents="false"&gt;
   &lt;stopWords&gt;
     &lt;stopWord&gt;le&lt;/stopWord&gt;
     &lt;stopWord&gt;la&lt;/stopWord&gt;
     &lt;stopWord&gt;les&lt;/stopWord&gt;
     ...
  &lt;/stopWords&gt;
&lt;/french&gt;
</programlisting> 
		<para>Sub-elements <sgmltag class="element">stopWord</sgmltag> contain stop words; moreover, the <sgmltag class="attribute">keepAccents</sgmltag> attribute defines whether the analyser shall delete diacritics or not. It is therefore an optional functionality of the word analyser for the French language.</para>
		<para>For each field, a word analyser must exist. This analyser must be directly specified, indirectly, be the default analyser of the document base or the default SDX analyser. NWe will examine how those indications must be used.</para>
		<section> 
		  <title>Directly specify a word analyser</title>
		  <para>Two attributes allow to directly specify which word analyser must be used: <sgmltag class="attribute">analyzerClass</sgmltag> and <sgmltag class="attribute">analyzerConf</sgmltag>. Those attributes can be associated to the <sgmltag class="element">sdx:field</sgmltag> element, and they identify the word analyser of the given field, otherwise they can be associated to the <sgmltag class="element">sdx:fieldList</sgmltag> element, they identify the default word analyser for the document base.</para>
		  <para>The <sgmltag class="attribute">analyzerClass</sgmltag> attribute indicates the class to be used as the analyser; such as described above, this class must extend the 
			 <ulink url="../../apidocs/fr/gouv/culture/sdx/search/lucene/analysis/Analyzer.html"><filename>fr.gouv.culture.sdx.search.lucene.analysis.Analyzer</filename></ulink> class and and it must obsiously be included in the Java CLASSPATH.</para>
		  <para>The <sgmltag class="attribute">analyzerConf</sgmltag> attribute indicates the location the configuration file for the word analyser. The attribute value is a URL, absolute or relative to the <filename>application.xconf</filename> file. If this attribute is not defined, the default configuration of the analyser is used. This attribute can also specify a configuration file if the word analyser is specified indirectly.</para>
		</section> 
		<section> 
		  <title>Specifying a word analyser indirectly</title>
		  <para>A word analyser may also be specified indirectly, which means by precising the language of the field content or of all document base fields. If the <sgmltag class="attribute">xml:lang</sgmltag> and <sgmltag class="attribute">variant</sgmltag> attributes are used with the elements <sgmltag class="element">sdx:field</sgmltag> (for a particular field) or <sgmltag class="element">sdx:fieldList</sgmltag> (as default value for all document base fields), SDX will attempt to find a suitable word analyser for this language.</para>
		  <para>The method used by SDX to find this analyser can be demonstrated through an example. Assuming the following definition&nbsp;:</para>
		  <programlisting>
&lt;sdx:field ... xml:lang="fr-CA" variant="qc"/&gt;

</programlisting> 
		  <para>The algorythme will be:</para>
		  <orderedlist>
			 <listitem>
				<para>If a class <filename>fr.gouv.culture.sdx.search.lucene.analysis.Analyzer_fr_CA_qc</filename> exists and if this class is available, it is used, otherwise, next step.</para>
			 </listitem><listitem>
				<para>If a class <filename>fr.gouv.culture.sdx.search.lucene.analysis.Analyzer_fr_CA</filename> exists and thos class is available, it is used, otherwise next step.</para>
			 </listitem><listitem>
				<para>If a class <filename>fr.gouv.culture.sdx.search.lucene.analysis.Analyzer_fr</filename> exists and this class is available, it is used, otherwise, next step.</para>
			 </listitem><listitem>
				<para>The default class 
				  <ulink url="../../apidocs/fr/gouv/culture/sdx/search/lucene/analysis/Analyzer.html"><filename>fr.gouv.culture.sdx.search.lucene.analysis.Analyzer</filename></ulink> is used.</para>
			 </listitem>
		  </orderedlist>
		  <para>If a <sgmltag class="attribute">xml:lang</sgmltag> attribute is used together with a <sgmltag class="attribute">analyzerClass</sgmltag> attribute, the latter is used to find the word analyser. The direct method is prioritary on the indirect one.</para>
		</section> 
		<section> 
		  <title>SDX default analyser</title>
		  <para>When no analyser is specified for a field or for a document base, SDX uses the 
			 <ulink url="../../apidocs/fr/gouv/culture/sdx/search/lucene/analysis/Analyzer_cn.html"><filename>fr.gouv.culture.sdx.search.lucene.analysis.DefaultAnalyzer</filename></ulink> analyser, which is a word analyser for the English language. This analyser is also used if a field indirectly declares an analyser that is not available with SDX.</para>
		</section> 
	 </section> 
	 <section> 
		<title>Ordering results</title>
		<para>Search results provided by SDX can be ordered accordinh to whatever field which is defined in the document base. Lists of terms of the fields are always given in alphabetic order. Both ordering operations performed by SDX try to take into account language of fields.</para>
		<para>If a <sgmltag class="element">sdx:field</sgmltag> element has a <sgmltag class="attribute">xml:lang</sgmltag> attribute and, possibly a <sgmltag class="attribute">variant</sgmltag> attribute, those information can be used to build a <emphasis>comparator</emphasis> (Java concept) which allows to correctly order results according to the language. Further information on those comparators and their use for ordering and text, see the 
		  <ulink url="http://java.sun.com/j2se/1.3/docs/api/java/text/Collator.html">java.text.Collator</ulink> Java class.</para>
		<para>Java virtual machines are usually released with several <emphasis>collator</emphasis> for various languages. Check the documentation of virtual machines for a list of available languages.</para>
	 </section> 
  </section> 

</webpage> 
