heino tsanangudzo pamusoro pekunyorwa kwemabhayiti maviri kuti anyorwe, pfungwa iyi ine zvimwe zvakanaka zvipi kunze kwekunyorwa?
Context:Byte pair encoding (BPE) kana digram coding inzira yakapfava uye yakasimba yekumanikidza data umo vaviri vanowanzoitika vepedyo byte yedata mune akateedzana vanotsiviwa ne byte isingaitike mukati mekateedzana. Tafura yokutsvaka yokutsiva inodikanwa kuvakazve mashoko apakuvamba. Algorithm yacho yakatanga kurondedzerwa pachena naPhilip Gage munyaya yaFebruary 1994 "A New Algorithm for Data Compression" muC Users Journal.  Imwe shanduko yemaitiro yakaratidza kuve inobatsira munzvimbo dzinoverengeka dzekugadzirisa mutauro (NLP), kune zvinoshandiswa senge tokenisation, sezvinoonekwa muGoogle's SentencePiece uye OpenAI's GPT-3. Pano, vavariro haisi yekumanikidza dhata, asi tokenisation yezvinyorwa mumutauro wakapihwa kuti ubudise kuteedzana kunochinja kwemashoko kubva kune yakatarwa-saizi dura remashoko ezviratidzo. Kazhinji, mashoko akawanda achanyorwa sechiratidzo chimwe chete, nepo mashoko asingawanzowaniki achanyorwa sechiratidzo chinotevedzana chezviratidzo zvishomanana, apo zviratidzo izvi zvinomiririra zvikamu zvemashoko zvine zvazvinoreva. Iyi shanduro yezvinyorwa muzviratidzo inogona kuwanwa nemisiyano yekunyorwa kwepairi byte, zvakadai semazwi mashoma.  Byte pair encoding inokodzera NLP mabasa nekuda kwekureruka kwayo uye nekukurumidza; BPE inoshanda zvakakwana kune tokenisation yematemu, haidi makuru ekuverenga pamusoro, uye inoramba yakafanana, ichiita kuti ive yakavimbika.
Chimwe chinoshandiswa pakunyorwa kwepairi byte kunze kwekumanikidza kungave kuita tokeni. Tokenization inogona kushandiswa muNatural Language Processing kunyorera zviratidzo zvemashoko kuti zvive nyore uye nekukurumidza mumabasa eNLP.