janodijk 2018-08-01+02:00 clarin.eu:cr1:p_1342181139640 CLARIN Netherlands
Resource https://webservices-lst.science.ru.nl/ucto/ Ucto Ucto Tokeniser v0.13 2011-03-27 https://webservices-lst.science.ru.nl/ucto/none yet published 2018-05-17 CLARIN-NLCLARIN in the Netherlands184.021.003NWOhttp://www.clarin.nlJan OdijkNational Coordinator
Utrecht, the Netherlands
j.odijk@uu.nlUiL-OTSUtrecht University20092015CLARIAH-CORECommon Lab Research Infrastructure for the Arts and the Humanities184.033.101NWOhttp://www.clariah.nlJan OdijkNational Coordinator
Utrecht, the Netherlands
j.odijk@uu.nlUiL-OTSUtrecht University20152018NetherlandsNL Ucto tokenizes text files: it separates words from punctuation, and splits sentences. This is one of the first tasks for almost any Natural Language Processing application. Ucto offers several other basic preprocessing steps such as changing case that you can all use to make your text suited for further processing such as indexing, part-of-speech tagging, or machine translation. The tokeniser engine is language independent. By supplying language-specific tokenisation rules in an external configuration file a tokeniser can be created for a specific language. Ucto comes with tokenization rules for English, Dutch, French, Italian, and Swedish; it is easily extendible to other languages. It recognizes dates, times, units, currencies, abbreviations. It recognizes paired quote spans, sentences, and paragraphs. It produces UTF8 encoding and NFC output normalization, optionally accepts other encodings as input. Optional conversion to all lowercase or uppercase. Ucto supports FoLiA XML. written language tool sentence splitting tokenisation Enriching Data Linguistics general linguistics syntax yes Dutchnld yes 20 21 yes Swedishswe yes 20 21 yes Russianrus yes 20 21 yes Spanishspa yes 20 21 yes Portuguesepor yes 20 21 yes Englisheng yes 20 21 yes Germandeu yes 20 21 yes Frenchfra yes 20 21 yes Italianita yes 20 21 Online available https://github.com/LanguageMachines/ucto not specified not specified POSIX not specified unknown icu not specified http://site.icu-project.org/design/cpp localDesktop libxml2 not specified https://pypi.org/project/libxml2-python/ localDesktop ticcutils not specified localDesktop libfolia not specified https://github.com/LanguageMachines/libfolia localDesktop command line interface local desktop graphical user interface web application web interface web service text PDF application/pdf text MS-Word application/msword utf8 text FoLiA text/folia+xml utf8 text text/plain ISO-8859-1 text text/plain ISO 8859-15 text text/plain text utf8 One Sentence per Line text/plain Discourse/Sentence Boundaries Orthography/Token text utf8 One Token per Line text/plain Discourse/Sentence Boundaries Orthography/Token text utf8 FoLiA text/folia+xml Discourse/Sentence Boundaries Orthography/Token GNU GPL 3.0 public https://spdx.org/licenses/GPL-3.0 0 EUR Antal van den Bosch
Nijmegen, the Netherlands
a.vandenbosch@let.ru.nl Center for Language and Speech Technology Radboud University Nijmegen https://www.ru.nl/clst/ Maarten van Gompel, Ko van der Sloot and Antal van den Bosch. 2017. Ucto: Unicode Tokenizer. version 0.9.6. Reference Guide. Technical Report, Jan 23, 2017. technical https://raw.githubusercontent.com/proycon/ucto/master/docs/ucto_manual.pdf eng Maarten van Gompel, Ko van der Sloot, Antal van den Bosch (2012). Ucto: Unicode Tokeniser. Reference Guide. ILK Technical Report 12-05. technical http://ilk.uvt.nl/downloads/pub/papers/ilk.1205.pdf eng readme user https://github.com/LanguageMachines/ucto/blob/master/README.md eng releaseNotes user https://github.com/LanguageMachines/ucto/releases eng issueTracker technical https://github.com/LanguageMachines/ucto/issues eng contIntegration technical https://travis-ci.org/LanguageMachines/ucto eng https://raw.githubusercontent.com/LanguageMachines/ucto/master/logo.svg CLARIN-NL <funder>NWO</funder> <url/> <Contact> <Person/> <Email/> <Organisation xml:lang="eng"/> </Contact> <Duration/> </Project> <Project> <name>CLARIAH-CORE</name> <title/> <funder>NWO</funder> <url/> <Contact> <Person/> <Email/> <Organisation xml:lang="eng"/> </Contact> <Duration/> </Project> <Creator> <Contact> <Person>Antal van den Bosch</Person> <Email/> <Organisation xml:lang="eng"/> </Contact> </Creator> <Creator> <Role> project lead </Role> <Contact> <Person> Antal van den Bosch </Person> <Address>Nijmegen, the Netherlands</Address> <Email> a.vandenbosch@let.ru.nl </Email> <Department>Center for Language and Speech Technology</Department> <Organisation> Radboud University Nijmegen </Organisation> <Url> https://www.ru.nl/clst/ </Url> </Contact> </Creator> <Creator> <Role> software developer </Role> <Contact> <Person> Maarten van Gompel </Person> <Address>Nijmegen, the Netherlands</Address> <Email> proycon@anaproy.nl </Email> <Department>Center for Language and Speech Technology</Department> <Organisation> Radboud University Nijmegen </Organisation> <Url> https://www.ru.nl/clst/ </Url> </Contact> </Creator> <Creator> <Role> software developer </Role> <Contact> <Person> Ko van der Sloot </Person> <Address>Nijmegen, the Netherlands</Address> <Department>Center for Language and Speech Technology</Department> <Organisation> Radboud University Nijmegen </Organisation> <Url> https://www.ru.nl/clst/ </Url> </Contact> </Creator> </SoftwareDevelopment> <TechnicalInfo> <ImplementationLanguage> <implementationLanguage>C++</implementationLanguage> <version>unknown</version> </ImplementationLanguage> </TechnicalInfo> <LRS> <Authentication>Yes. Before tool use, please register at https://webservices-lst.science.ru.nl/register.</Authentication> <Description><Description>Ucto</Description></Description> <ToolTasks> <toolTask>sentence splitting</toolTask> <toolTask>tokenisation</toolTask> </ToolTasks> <Input> <inputType>text</inputType> <Schema><schemaname>PDF</schemaname></Schema> <MimeType><MimeType>application/pdf</MimeType></MimeType> </Input> <Input> <inputType>text</inputType> <Schema><schemaname>MS-Word</schemaname></Schema> <MimeType><MimeType>application/msword</MimeType></MimeType> </Input> <Input> <characterEncoding>utf8</characterEncoding> <inputType>text</inputType> <MimeType><MimeType>text/plain</MimeType></MimeType> </Input> <Input> <characterEncoding>ISO-8859-1</characterEncoding> <inputType>text</inputType> <MimeType><MimeType>text/plain</MimeType></MimeType> </Input> <Input> <characterEncoding>ISO 8859-15</characterEncoding> <inputType>text</inputType> <MimeType><MimeType>text/plain</MimeType></MimeType> </Input> <Output><outputType>text</outputType> <characterEncoding>utf8</characterEncoding> <Schema><schemaname>One Sentence per Line</schemaname></Schema> <MimeType><MimeType>text/plain</MimeType></MimeType> <AnnotationType> <AnnotationType>Discourse/Sentence Boundaries</AnnotationType> <AnnotationType>Orthography/Token</AnnotationType> </AnnotationType> </Output> <Output><outputType>text</outputType> <characterEncoding>utf8</characterEncoding> <Schema><schemaname>One Token per Line</schemaname></Schema> <MimeType><MimeType>text/plain</MimeType></MimeType> <AnnotationType> <AnnotationType>Discourse/Sentence Boundaries</AnnotationType> <AnnotationType>Orthography/Token</AnnotationType> </AnnotationType> </Output> <Output><outputType>text</outputType> <characterEncoding>utf8</characterEncoding> <Schema><schemaname>FoLiA</schemaname></Schema> <MimeType><MimeType>text/xml</MimeType></MimeType> <AnnotationType> <AnnotationType>Discourse/Sentence Boundaries</AnnotationType> <AnnotationType>Orthography/Token</AnnotationType> </AnnotationType> </Output> <ActualParameters><!--0-1 --> <ActualParameter><!--1 - unbounded --> <ActualParameterName>project</ActualParameterName> <ActualParameterValue>new</ActualParameterValue> </ActualParameter> <ActualParameter><!--1 - unbounded --> <ActualParameterName>input</ActualParameterName> <ActualParameterValue>self.linkToResource</ActualParameterValue> </ActualParameter> <ActualParameter><!--1 - unbounded --> <ActualParameterName>lang</ActualParameterName> <ActualParameterValue>self.linkToResourceLanguage</ActualParameterValue> </ActualParameter> </ActualParameters> <LRSMapping> <LRSParameterName>input</LRSParameterName> <ActualParameterName>untokinput_url</ActualParameterName> </LRSMapping> <LRSMapping> <LRSParameterName>lang</LRSParameterName> <ActualParameterName>untokinput_language</ActualParameterName> </LRSMapping> </LRS> <LRS> <Authentication>Yes. Before tool use, please register at https://webservices-lst.science.ru.nl/register.</Authentication> <Description><Description>Ucto</Description></Description> <ToolTasks> <toolTask>sentence splitting</toolTask> <toolTask>tokenisation</toolTask> </ToolTasks> <Input> <characterEncoding>utf8</characterEncoding> <inputType>text</inputType> <Schema><schemaname>FoLiA</schemaname></Schema> <MimeType><MimeType>text/folia+xml</MimeType></MimeType> </Input> <Output><outputType>text</outputType> <characterEncoding>utf8</characterEncoding> <Schema><schemaname>One Sentence per Line</schemaname></Schema> <MimeType><MimeType>text/plain</MimeType></MimeType> <AnnotationType> <AnnotationType>Discourse/Sentence Boundaries</AnnotationType> <AnnotationType>Orthography/Token</AnnotationType> </AnnotationType> </Output> <Output><outputType>text</outputType> <characterEncoding>utf8</characterEncoding> <Schema><schemaname>One Token per Line</schemaname></Schema> <MimeType><MimeType>text/plain</MimeType></MimeType> <AnnotationType> <AnnotationType>Discourse/Sentence Boundaries</AnnotationType> <AnnotationType>Orthography/Token</AnnotationType> </AnnotationType> </Output> <Output><outputType>text</outputType> <characterEncoding>utf8</characterEncoding> <Schema><schemaname>FoLiA</schemaname></Schema> <MimeType><MimeType>text/folia+xml</MimeType></MimeType> <AnnotationType> <AnnotationType>Discourse/Sentence Boundaries</AnnotationType> <AnnotationType>Orthography/Token</AnnotationType> </AnnotationType> </Output> <ActualParameters><!--0-1 --> <ActualParameter><!--1 - unbounded --> <ActualParameterName>project</ActualParameterName> <ActualParameterValue>new</ActualParameterValue> </ActualParameter> <ActualParameter><!--1 - unbounded --> <ActualParameterName>input</ActualParameterName> <ActualParameterValue>self.linkToResource</ActualParameterValue> </ActualParameter> <ActualParameter><!--1 - unbounded --> <ActualParameterName>lang</ActualParameterName> <ActualParameterValue>self.linkToResourceLanguage</ActualParameterValue> </ActualParameter> </ActualParameters> <LRSMapping> <LRSParameterName>input</LRSParameterName> <ActualParameterName>foliainput_url</ActualParameterName> </LRSMapping> <LRSMapping> <LRSParameterName>lang</LRSParameterName> <ActualParameterName>foliainput_language</ActualParameterName> </LRSMapping> </LRS> </ClarinSoftwareDescription> </Components> </CMD>