I-YTsaurus iyinkundla yesitoreji esabalalisiwe nokucubungula idatha enkulu esekelwa imodeli ye-MapReduce.
Ezinsukwini ezithile ezedlule I-Yandex ivuliwe ngesinye esimenyezelwe ukuvulwa kwekhodi yomthombo yesikhulumi se-Ytsauru, esetshenziselwa ukugcinwa okusatshalaliswa kanye nokucutshungulwa kwenani elikhulu ledatha, esekela ukukhohliswa kwedatha kusetshenziswa i-MapReduce paradigm, injini yombuzo ye-SQL, isistimu yefayela esabalalisiwe kanye nesitoreji se-NoSQL ngefomethi yenani elingukhiye.
I-YTsaurus esetshenzisiwe kwingqalasizinda ye-Yandex ukusebenzisa kahle amandla ekhompuyutha ama-supercomputers yenkampani Inkundla ingafinyelela kumaqoqo ama-node angaphezu kuka-10, ahlanganisa amaphrosesa afika esigidini nezinkulungwane zama-GPU (emisebenzini yokufunda komshini).
Iziqukathi ezingazodwa ezisebenza kumaseva aphathekayo zingasetshenziswa njengamayunithi eqoqo. Isitoreji singaqukatha ama-exabytes edatha atholakala kumidiya ehlukahlukene njengama-hard drive, ama-SSD, i-NVME, ne-RAM.
Iqoqo lisekela ukungezwa okuguquguqukayo nokususwa kwama-node, ukuphindaphinda (akukho iphuzu elilodwa lokwehluleka), ukuphindaphinda okuzenzakalelayo, ukuthuthukiswa kwesofthiwe yeqoqo elisebenzayo, kanye nokubuyisela kabusha okuzenzakalelayo uma kwenzeka ukwehluleka kwenodi.
Izinhlobo ezintathu zamaqoqo ziyasekelwa: hlanganisa amaqoqo (ngokucubungula okuhambisanayo okukhulu kwedatha enkulu usebenzisa imisebenzi ye-MapReduce), amaqoqo amathebula e-pivot kanye nesitoreji senani elingukhiye, namaqoqo asabalaliswe ngokwendawo.
Isevisi esekwe kungxenyekazi ingahlinzeka ngezindlela zokugcina nokucubungula idatha yamashumi ezinkulungwane zabasebenzisi. Izinhlelo zokusebenza ezijwayelekile ze-YTsaurus ku-Yandex zihlanganisa ukugcina ulwazi olumayelana nabasebenzisi benethiwekhi yezikhangiso, amamodeli wokufunda womshini wokuqeqesha, ukwenza inkomba yosesho, nokwakha indawo yokugcina idatha yamasevisi afana ne-Yandex Taxi. , Ukudla, i-Lavka nokudiliva
Ezimweni eziyisisekelo zokusetshenziswa kushiwo:
- Ukucutshungulwa kwenqwaba: I-MapReduce kanye ne-SPYT (i-Apache Spark njengenjini yokwenza ikhompuyutha kudatha ku-YTsaurus) ukuze kucutshungulwe idatha ehlelekile nesakhiwe kancane: amarekhodi noma imisebenzi yezezimali.
- Ukuhlaziywa kwe-Ad hoc: Imibuzo esheshayo nge-CHYT (iqembu lamaseva e-ClickHouse ku-YTsaurus compute nodes) ngaphandle kokukopisha idatha ohlelweni oluhlukile lwezibalo. I-ODBC ne-JDBC enekhono lokuxhuma i-BI ukuze ibonwe.
- Imisebenzi ye-OLTP: Umsebenzi wesikhathi sangempela onesitoreji senani elingukhiye: isibonelo, isitoreji sephrofayela yomsebenzisi, isibonisi sesikhangiso, noma ukucubungula ukusakaza.
- Ukufunda ngomshini: Lawula amaqoqo e-GPU ukuze uqeqeshe amamodeli anezigidigidi zamapharamitha.
- Isitoreji semininingwane ye-Meta: Ukugcinwa kokuthengiselana kwe-metainformation kanye nesevisi ethembekile yezingidi ezisabalalisiwe.
- Ukudalwa kwezindawo zokugcina idatha kanye ne-ETL yokucubungula idatha yezigaba eziningi kusetshenziswa amathuluzi ajwayelekile: i-Apache Spark, SQL, MapReduce.
Nakuba ngasohlangothini lwezici ezibalulekile zezakhiwo, kushiwo okulandelayo:
- Isistimu yefayela esabalalisiwe kanye nesitoreji solwazi se-Cypress esibekezelela iphutha esisekelweni sesihlahla.
- Isihleli sekhompuyutha esabalalisiwe esisekelwa imodeli ye-MapReduce, kanye nemisebenzi eyisisekelo ethuthukisiwe.
- Ukukala okuvundlile kokusebenza kwe-IT.
- Ukuhlukaniswa kwezinsiza zekhompiyutha kanye nokwenzeka kokwaba izinsiza ezithile zekhompiyutha (i-CPU, i-GPU, i-RAM) ngezilinganiso ezihlukene.
- Ama-PivotTables okudala isitoreji se-OLTP, usekelo lwesitoreji esisekelwe ku-MVCC, ukuthengiselana, ikhono lokususa idatha ngemva kokuphelelwa yisikhathi, kanye nemigqa yemilayezo yokusakaza idatha ku-PivotTables.
- I-API nemitapo yolwazi yezilimi zokuhlela C++, Python, Java, Go.
- Isixhumi esibonakalayo sewebhu sabasebenzisi nabalawuli esisekela ukuzulazula ngesitoreji esinjengesihlahla.
Okokugcina uma unentshisekelo yokwazi kabanzi ngakho, kufanele wazi ukuthi ikhodi yephrojekthi ibhalwe ku-C/C++ futhi ivuliwe ngaphansi kwelayisensi ye-Apache 2.0. Ungahlola imininingwane ku- isixhumanisi esilandelayo.
El Indawo yokugcina izinto zeGitHub iqukethe ikhodi yeseva ye-YTsaurus, uhlaka lokuphakela olusetshenziswa ama-k8s, isixhumi esibonakalayo sewebhu ohlelweni, kanye nama-SDK eklayenti ezilimi zokuhlela ezidumile ezifana ne-C++, Java, Go, nePython.