Chii Corpus zvemutauro?

Mashomanana emakore apfuura kuti michina yacho mitauro tsvakurudzo, masayendisiti aigona chete kufungidzira. Basa rakaitwa noruoko, kwava vazhinji vadzidzi, pane nouchapupu zvingangoratidza zvikanganiso "kusava", uye Chikurukuru - izvi zvose akatora refu, yakareba.

With kukura zvigadzirwa kombiyuta yava zvichibvira kuita ongororo murayiro unokosha nokukurumidza, uye nhasi rimwe vaizotibata zvikuru mirayiridzo kudzidza mutauro uri corpus zvemutauro. Its chikuru chepfungwa ndiko kushandisa yakawanda chinyorwa mashoko, mashoko mugomba chete Database, neimwe nzira uye akadana waivapo muviri.

To musi, pane zvivako zvakawanda zvakasikwa chete siyana zvinangwa maererano siyana mitauro zvinhu zvakatevedzana kubva mamiriyoni makumi emabhiriyoni lexical dzakabatana. mirayiridzo iri kuzivikanwa sezvo aizobudirira uye kunoratidza kukosha mberi kurutivi kushanda nokutsvakurudza zvinangwa. Nyanzvi, imwe nzira kana imwe pakubata dzinongoitika mutauro, Zvinokurudzirwa tinyatsoziva muviri magwaro kupfuura panguva inokosha pamwero.

History of corpus zvemutauro

Paumbwe ichi Muitiro imhaka pakusikwa United States pana Brown muviri pakutanga 60-ies wekupedzisira remakore. Kuunganidza kunosanganisira magwaro ose miriyoni 1 shoko siyana, uye nhasi muviri kukura uku kwaizova zvachose uncompetitive. Izvi kukonzerwa nokufamba kukura noruzivo rwemakombiyuta, uyewo kukura zvinodiwa itsva tsvakurudzo kupfuma.

Munguva 90s corpus zvemutauro yakabuda munzvimbo zvizere uye akazvimirira kurangwa, dzakaunganidzwa magwaro vave akambochera uye chiratidzo yakawanda. Iyi yakanga akasikwa Somuenzaniso, zviratidzo British National Corpus 100 miriyoni.

With kukura munharaunda iyi wemitauro, rwezuva mavhoriyamu vari kungoramba (uye kusvika mabhiriyoni duramashoko dzakabatana), uye marongerwo iri kuramba ichiva siyana. Pari, Internet nzvimbo inogona kuwanikwa zvitunha akanyorwa uye mutauro, mitauro, uye unyanzvi kana zvedzidzo mabhuku kudzidza-zvinangwa, uyewo dzimwe mhando dzakawanda.

Ndezvipi dzimba

Muviri mhando mumuviri zvemutauro anogona kupiwa nezvikonzero zvinoverengeka. Intuitively, panobva kupatsanura kunogona mutauro wenyaya (Russian, German), kuti mapindiro muoti (rakavhurika bhuku, akavharwa, zvokutengeserana), kuti Genre pakati anokonzera zvinhu (ingano, vhidhiyo, zvedzidzo, nhau).

Interesting nzira zvinowanisa zvinhu zviri kutaurwa mutauro. Sezvo kurekodha nemaune yokutaura kwakadaro kusika rakagadzirwa nevanhu zvakatipoteredza yevakapindura, uye kunoguma zvinhu vasingagoni kunzi "kwatangwa", ano corpus zvemutauro kwabva dzimwe nzira. Anozvipira ari akashongedzerwa aine microphone, uye masikati yakabereka zvinyorwa kukurukurirana zvose, pazviri chikamu. Vanhu kunhivi Chokwadi, murege kuziva kuti mukufamba kwezuva nezuva kunoita kuti pave nesayenzi.

Akazopiwa zvinyorwa mudura Database uye vari pamwe akadhindwa chinyorwa inopangidza mhando. Saka, zvinova zvinoita markup aifanira uvhure waingoita zuva nokutaura dzimba.

kunyoresa

Pose pazvinobvira kushandiswa mutauro, uye zvichida kushandisa zvivako magwaro. Nzira kushandisa Hull mune zvemutauro angava:

• Kutanga purogiramu kusarudza kiyi, iri zvinowanzoshandiswa zvematongerwo enyika uye mabhizimisi kuti arambe akanaka uye kunaka mabasa vavhoti uye vatengi, ukuwo.
• Connection mashoko ezvinhu maduramazwi uye vashanduri kunatsiridza vaita.
• A-siyana tsvakurudzo mabasa kuti dzinobatsira kunzwisisa mutauro Unit, nhoroondo yacho kukura uye Kufembera kuchinja remangwana.
• Development mashoko acho gadziriro inobva morphological, syntactic, kupindirana kwemuvanzrira wezvirevo nemabviro uye zvimwe zvinhu.
• Optimization of mitauro yakasiyana hurongwa uye nevamwe.

Use zvivako

zvakafanana zvekuvaka inowanikwa ane nezvekumwe search engine, uye kunoita inodzidzisa kupinda shoko kana mubatanidzwa mashoko kutsvaka ruzivo chawo. Tisingatauri vanoumba chaiwo mubvunzo anogona kushandisa kunowedzera shanduro, iyo anobvumira kuwana mamwe mashoko mashoko chero kwomutauro zvinodiwa.

Tsvakiridzo chawo chingava:

• nhengo imwe boka zvikamu kutaura;
• zvivakamutauro zviri;
• semantics;
• manyorero uye manzwiro coloring.

Unogonawo chokubatanidza kutsvaka nzira kuti kutevedzana mashoko Somuenzaniso, kuwana nokuwanikwa dzose rechiito iri chipo nerongo, chokutanga munhu mumwe chete, hunouya mushure rokuti "mu" uye rokuti ari accusative nyaya. The Kugadzirisa rakadaro nyore basa anotora User mumasekonzi mashoma uye kunoda vashoma mbeva the clicks ari kutaurwa minda.

Muitiro pakusika

The kutsvaka pachayo inogona kuitwa pamusoro subcorpus zvose uye rimwe zvakananga akasarudzwa, zvichienderana nezvinodiwa kuwana imwe chinangwa:

1. Danho rokutanga kutsanangura Magwaro vanoumba hwaro nyaya. Zvinangwa zvinoshanda, zviri anowanzoshandiswa journalistic, nhau nyaya, online mashoko. Zvokutsvakurudza chirongwa ndiko kushandiswa akasiyana-siyana pasuru mifananidzo, asi rugwaro kunofanira kusarudzwa maererano zvimwe zvavaibvumirana.
2. The zvichiguma muunganidzwa magwaro pasi pretreatment, pane kurangwa zvikanganiso, kana ani, yakagadzirwa rondedzero rugwaro Bibliographic uye mamwe-mitauro.
3. Panobviswa zvose non-zvakawedzerwa mashoko: paachabvisa Graphics, mifananidzo, namatafura.
4. Ane hurumende ichiisa zviratidzo, idzo yemanyorero kutaura, nokuti mamwe kugadzira.
5. Pakupedzisira, yakatakura morphological, syntactical uye dzimwe nyora vakawana muzvizhinji kuti zvinhu.

Mugumisiro kwema zvose kunoitwa syntactic chivako pamwe vakagovera mairi ane muzvizhinji kuti zvinhu, chimwe nechimwe chinozivikanwa mukukurukurirana, zvivakamutauro uye, mune zvimwe zviitiko, kupindirana kwemuvanzrira wezvirevo nemabviro hunhu.

Zvinetso mukusika zvivako

Zvakakosha kunzwisisa kuti hazvina kukwana kuti kuisa pamwechete kuva mashoko kana mitsetse muviri. Kudivi rimwe, dzakaunganidzwa magwaro vanofanira kuva pakati nepakati, kureva, anomiririra mapoka akasiyana magwaro mune zvimwe zvikuru. Kune vamwe - zviri yakakomberedzwa inofanira munzvimbo dzokuzororera dzaiva munzira chaiyo.

Dambudziko rokutanga kugadziriswa chibvumirano: somuenzaniso, ari muunganidzwa inosanganisira 60% kubva mumabhuku, 20% anonzi, mumwe chikamu anopiwa chiratidzo cherudo rakataurwa mutauro, mutemo, mabasa dzesayenzi, etc. akakwana yokubika mikwende muviri nhasi hakuwaniki kunyorwa ...

mubvunzo wechipiri, maererano kugutsikana marongerwo, kugadzirisa nyore. Kune panorongwa uye algorithms yaishandiswa otomatiki kupenda magwaro, asi havazivi kupa mugumisiro akakwana, anogona kukonzera disruptions uye inoda Chinyorwa rework. Mikana uye nematambudziko pakubata dambudziko iri vanorondedzerwa zvakadzama pepa V. P. Zaharova pamusoro corpus zvemutauro.

Text markup iri rakatangwa panguva dzakasiyana siyana, izvo isu zvinyore pasi.

morphological tagging

Kubva kuchikoro, tinoyeuka kuti mumutauro wokuRussia, pane dzakasiyana-siyana kutaura, uye mumwe nomumwe wavo ane pachake zvachiri. Somuenzaniso, shoko rechiito rine Mapato ose mafungiro uye nguva nenguva iyo hakuna pazita. chizvarwa mutauri pasina kumbozengurira kupera emazita uye Conjugate verbs, asi unotara muviri mamiriyoni 100. zviratidzo remaoko hakushandi. Zvisinei zvose akavhiyiwa zvakafanira zvinogona kuuraya kombiyuta, uyu anofanira kudzidziswa.

Morphological tagging, kombiyuta anofanira "kunzwisisa" Shoko rimwe nerimwe sezvo imwe mukukurukurirana ane zvimwe girama zvinhu. Sezvo Russian (uye mumwe mutauro) unoshanda dzinoverengeka dzose mitemo, zvinokwanisika kuti avake otomatiki nzira nokuda morphological pachisara bhizimisi mumotokari wandei algorithms. Zvisinei, pane kunze kuushe, uyewo siyana complicating zvinhu. Somugumisiro, mambure kombiyuta kuongorora nhasi kumbonaka, uye kunyange 4% kukanganisa unobereka ane ukoshi 4 mln. Mashoko pamusoro pomuviri 100 miriyoni. Zviyero, rinoda Chinyorwa rework.

Detailed bhuku rinotsanangura dambudziko Zaharova V. P. "Corpus zvemutauro".

syntactic Musumo

Parsing kana parsing - muitiro kuti anosarudza ukama mashoko mumutsetse. Kushandisa mamwe algorithms zvinobvira kuziva mashoko enyaya, Predicate, kuwedzererwa, akawanda vachichinjana kutaura. Tsvakai kuti ndeapi mashoko chikuru urongwa, uye iyo - kubatsirwa, isu zvinobudirira vanokwanisa kusvina mashoko kubva murugwaro uye kudzidzisa muchina kuti ape papiwa kutsvaka chikumbiro chete mashoko anonakidza nesu.

Zviya, ano kutsvaka inoshandisa ichi kupa nezvedzimwe nhamba panzvimbo zvinyorwa refu semhinduro akakodzera queries akadai "sei macalorie akawanda apuro" kana kuti "kure Moscow kuna St. Petersburg." Zvisinei, kunzwisisa kunyange zvinokosha kwacho yakarondedzerwa vanofanira kubvunza "Nhanganyaya Corpus zvemutauro" kana zvimwe zvinhu Tutorial.

kupindirana kwemuvanzrira wezvirevo nemabviro markup

The semantics kweshoko - iri, zviri nyore, zvazvinoreva. Zvikuru kushanda kusvika kune kupindirana kwemuvanzrira wezvirevo nemabviro kuongorora shoko Attribution Tags, uchiratidza Kuva kwake kutaridza kupindirana kwemuvanzrira wezvirevo nemabviro mhando uye Subcategories. mashoko kwakadaro kwakakosha optimizing algorithms ongorora chinyorwa matauriro, otomatiki summarization uye dzimwe mabasa nzira corpus zvemutauro.

Kune akawanda "mudzi" muti, anomiririra husingaoneki shoko navanopinda chaizvo semantics. Sezvo bazi remuti nodes vari kuumbwa, rine zvakawanda chaiwo lexical okunze. Somuenzaniso, shoko rokuti "chisikwa" anogona chokuita nounhu hwakadai so "munhu" uye "mhuka". Shoko rokutanga acharamba kubazi kunyika dzakasiyana ounyanzvi hwokugadzira, ukama mashoko, nyika, uye wechipiri - pamusoro makirasi uye mhando dzemhuka.

Kushandiswa mashoko acho nehurongwa

Nzvimbo kushandiswa corpus zvemutauro akafukidza minda basa siyana. Housings inoshandiswa achigadzirira uye kururamiswa nemaduramazwi, kusika zvoga shanduro mugadziriro, annotating, wokuwanazve chokwadi, kuziva matauriro uye mamwe acho kugadzira.

Uyezve, zvinhu zvakadai uchipinda kushandiswa kudzidza mitauro nyika uye zvigadziriswe kuti kushanda mutauro zvavo. Access kune huru zvemaBhaibheri pre-akagadzirira mashoko hwezvekukurukurirana nokukurumidza uye chinoratidza kudzidza dzinoitwa kukura mitauro, uye akatsiga inotungamirwa neologisms kutaura nokukurumidza shanduko anokoshesa lexical dzakabatana uye nevamwe.

Sezvo basa yakadai yakawanda date kunoda kushandisa michina, nhasi pane pedyo kusakarongeka pakati kombiyuta uye corpus zvemutauro.

Russian National Corpus

nyaya iyi (rakapfupikiswa NKRYA) unosanganisira dzinoverengeka subcorpus, achibvumira kushandisa ruzivo kuti siyana mabasa.

Zvinhu zviri Database vakakamurwa NKRYA:

• kune mabhuku vezvenhau '90s uye 2000s, zvose mudzimba uye vatorwa;
• pokunyora kutaura;
• aktsentologicheski akacherekedza magwaro (i.e., mavanga mupfungwa);
• bazi remutauro kutaura;
• nhetembo;
• Zvinhu pamwe syntactic uye nezvimwe nyora.

Mashoko Hurongwa anosanganisirawo Subcorpus pamwe inofambirana shanduro mabasa kubva Russian kupinda Chirungu, mimwe mitauro German, French uye vazhinji (uye vakutarisire).

Uyewo ari Database pane chikamu chenhau magwaro, rinomiririra akanyorwa rwokutaura Russian dzakasiyana nguva yaro kukura. Panewo muviri kudzidzisa, zvinogona kubatsira kuti zvizvarwa wokumwe mukudzidzira mutauro weRussia.

Russian National Corpus unotora mamiriyoni 400 lexical dzakabatana, uye nenzira zhinji pamberi chinokosha chikamu mitauro Europe miviri.

tarisiro

Chokwadi vachida kuzivikanwa uyu muitiro ndiko Kuwanikwa vachivimbisa rabhoritari corpus zvemutauro mu Africa Russian, uyewo dzimwe. Nokushandiswa uye kutsvakurudza yechinangwa ichi ruzivo uye kutsvaka pfuma rinobatanidza dzimwe nzvimbo mumunda mukuru ruzivo, mubvunzo-kupindura hurongwa, asi zvataurwa pamusoro apa.

Pakuzadzisa corpus zvemutauro akafanotaurwa zvese, kubvira hwokugadzira uye maererano Implementation pamusoro algorithms itsva kuti optimize kwemasayendisiti yokutsvaka uye processing ruzivo, makombiyuta masimba, zvakawanda RAM, uye kuti mutengi, nekuti vanoshandisa ndivo vachiwedzera nzira kushandisa ichi mhando zvekuvaka zvavo zvezuva nezuva upenyu uye basa.

In mhedziso

Pakati pezana remakore rapfuura muna 2017 sokuti ramangwana kure, apo musere musere vofamba zvakasikwa uye marobhoti kuita basa rose vanhu. Kutaura zvazviri, sainzi rakazara "mavara machena" uye kuita zvaari kupindura mibvunzo vanhu kwemazana emakore bopoto. Mibvunzo kushanda mutauro pano kugara nzvimbo yapamberi, uye cabinet uye computational zvemutauro zvinogona kutibatsira kupindura navo.

Processing makuru mashoko seti vanogona kuzviona mapatani, kare dzisingambosvikiki, kufanotaura kuvandudzwa dzakananga mutauro zvinhu kuti chamber paumbwe mashoko nguva inenge zvechokwadi.

On inoshanda sandara, munyika yose zviruva zvinogona kuonekwa Somuenzaniso, sezvo vanokwanisa mudziyo kuongorora pachena manzwiro - Indaneti iri nguva achiudza nezuva siyana mumagwaro vakasikwa chaiye vanoshandisa: comments uye ongororo, uye nyaya, uye dzimwe mhando kutaura zvakawanda.

Uyezve, kushanda miviri chinowedzera kukura chete Hardware, kuti urimowo mashoko acho, isu vanoziva basa "Google" kana "Yandex", muchina rokushandura, zvemagetsi maduramazwi.

Tinogona vanoti nechivimbo kuti corpus zvemutauro anoita chete wokutanga matanho, uye munguva yemberi iri pedyo richafara.