Idatha enkulu, isoftware yasimahla kunye nomthombo ovulekileyo: Izicelo ezikhoyo

Idatha enkulu ngumqondo wobuchwephesha onxulumene nolawulo lweedatha ezinkulu, yakhiwe kwaye ayakheka, ekungokunje ilawulwa ngamashishini amakhulu, iteknoloji, inzululwazi kunye namacandelo karhulumente.

Nangona xa uthetha malunga Idatha enkulu, ayisiyiyo eyona datha ibalulekileyo, kodwa yeyiphi imibutho eyenzayo ngedatha. Ukusukela kwiDatha eNkulu, itekhnoloji enxulumene nayo, inokuzihlalutya ukuze ifumane izimvo ezikhokelela ekuthathweni kwezigqibo okungcono, iintshukumo kunye nezicwangciso. Kule nkalo, Isoftware yasimahla (i-SL) kunye noMthombo ovulekileyo (i-CA) banegalelo elikhulu kule teknoloji, kuba uninzi lwezicelo eziphuhlisiweyo zenziwe kule fomathi yophuhliso.

Idatha enkulu kunye neSoftware yasimahla

Kwabo banobugcisa kubugcisa, sele isaziwa kakuhle loo nto Isoftware yasimahla, imodeli yayo yophuhliso, ifilosofi yayo, isekwe ekwenzeni iitekhnoloji, ngakumbi iimveliso zesoftware, ezinokuthi zisetyenziswe ngokukhululekileyo, zilungiswe kwaye zihanjiswe. Kwaye uMthombo ovulekileyo uyinto ebalulekileyo kuphuhliso lwesoftware yasimahla, kuba ijolise kwizibonelelo eziluncedo zolu phuculo olunamandla ngakumbi kunakwimigaqo yenkululeko yemveliso kunye Abemi.

Ke ngoko, ngelixa I-SL / CA inegalelo ngeendlela zokwenza iDatha eNkulu, Idatha enkulu iyakugcwalisa ngokungangqalanga, hayi kuphela inzuzo yokunyuka okukhawulezayo kophuhliso lobuchwephesha, kodwa nakwinkululeko yokufikelela kulwazi oluza neDatha eNkulu.

Ithini idatha enkulu?

Umxholo

Ngenye yee-greats zeSoftware kunye nophuhliso lwetekhnoloji, IBM, Idatha enkulu yile:

«... itekhnoloji evule iingcango zendlela entsha yokuqonda kunye nokwenza izigqibo, esetyenziselwa ukuchaza izixa ezikhulu zedatha (eyakhiwe, engacwangciswanga kunye neyakhiwe kancinci) enokuthatha ixesha elide kwaye ibize kakhulu ukuyifaka isiseko sedatha ehambelana nohlalutyo.

Injongo

Idatha enkulu, itekhnoloji yayo, yazalwa ngenjongo yokugubungela yonke into ebonakalayo yohlalutyo lwedathaOko kukuthi, ukugubungela kokubini okukhoyo nokusonjululwa ngetekhnoloji yangoku neyahlukileyo, kunye nokungasombululwa bubuchwephesha obukhoyo, ukugcinwa kunye nolawulo lwemiqulu emikhulu yedatha ezineempawu ezithile.

Iinkcukacha

Idatha yeBhidi ibamba inani ledatha esihlala sichazwa zezi mpawu zilandelayo:

• Umqulu: Ubungakanani bedatha evela kwimithombo emininzi.
• Isantya: Isantya ekuthi ngaso idatha evela kwimithombo emininzi ifike kwaye ilawulwe.
• Ezahlukeneyo: Ifomathi yedatha ehlalutyiweyo evela kwimithombo emininzi.

Ndiyathetha Imiqulu yedatha edla ngokwenziwa ngeDatha eLungisiweyo, eyakhiwe ngokungagqibelelanga nengacwangciswanga, kwaye ziphathwe ngobuninzi obukhulu obuhlala buchazwa ngezimaphambili zamanani aphezulu, anje nge: Tera, Peta okanye i-Exa, phakathi kwabanye.

Kuzo zonke iintlobo zemithombo yolwazi, njenge-Intanethi (Iinethiwekhi zentlalo, iMithombo yeDijithali, iiWebhusayithi kunye neeDatha), Amaqela (Iifowuni eziphathwayo, abadlali beMultimedia, iiNdawo zokuBekwa, iiNzwa zedijithali zombuso kunye nezamashishini, phakathi kwabanye) kunye Imibutho (Ezabucala nezikaRhulumente, ezoRhwebo, uRhulumente noLuntu).

Ukubaluleka

Yintoni eyenza ukuba iDatha eNkulu itekhnoloji eluncedo kangaka yeMibutho (Ezabucala nezikaRhulumente, ezoRhwebo, uRhulumente noLuntu), yinto yokuba ibonelela ngolwazi oluxabisekileyo ehlala isebenza njengempendulo echanekileyo ethembekileyo kwimibuzo engakhange ibuzwe nokubuzwa yeemeko ezithile okanye iingxaki. Oko kukuthi, ukusebenziseka kwayo kuhlala kubonakala kwizinto ezihlala zivela kulwazi olufanayo oluqokelelweyo nolulawulwayo.

Ukulungiswa kwemiqulu emikhulu yolwazi kwenza ukuba kube lula ukuba idatha eqhutyiweyo yenziwe okanye ivavanywe ngendlela efanelekileyo. okanye icacisa, ethathwa njengefanelekileyo ngumlawuli wayo. Oku kuvumela imibutho esebenzisa i-Big Data ukuba ikwazi ukufumanisa iingxaki ngendlela eqondakala ngakumbi.

Ukuqokelelwa kwemithamo emikhulu yedatha kunye nohlalutyo lwayo olulandelayo ukukhangela imeko ngaphakathi kuzo kuvumela iMibutho ukuba isebenze ngakumbi kwaye isebenze, ngokuhamba ngokukhawuleza okukhulu, ngokutyibilikayo kwaye ngexesha elifanelekileyo ngaphezulu kwazo. Ukongeza, ivumela ukuba bashenxise indawo ezinengxaki ngaphambi kokuba iingxaki zibaphathe, zibangele ukuba baphulukane nezibonelelo, udumo okanye inkxaso.

Inzuzo

Idatha enkulu inceda Imibutho ukuba ilawule idatha yayo ngcono, oku kukhokelela ekuchongweni kwamathuba amatsha aqinisekileyo okanye okuvelisa amalungu awo (abathengi okanye abemi). Kwaye oku, kukhokelela kwizenzo ezikrelekrele nezisebenza ngokukuko, ukonga iiyure / umsebenzi kunye nemali, zihlala ziguqulela kulonwabo kuye wonke umntu obandakanyekayo. Xa kusetyenziswa i-Big Data, ixabiso lihlala longezwa kwimisebenzi eyenziwa ngezi ndlela zilandelayo:

• Ukunciphisa iindleko: Ekugcinweni nasekulawulweni kwenani elikhulu ledatha.
• Ukuncitshiswa kwexesha: Ukusebenza ngakumbi nangokufanelekileyo ekuthathweni kwezigqibo.
• Iimveliso ezintsha kunye neenkonzo: Ngokwazi ukulinganisa kunye nokuqikelela iimfuno kunye neengxaki zabasebenzisi (abathengi kunye / okanye abemi), ukoneliseka kwabo kuyanda.

Izinzuzo

Ukusetyenziswa kakuhle kwedatha enkulu ihlala ikwazi ukumisela oyena nobangela wokusilela, iingxaki kunye neziphene phantse ngexesha lokwenyani. Nangona kunjalo, kukuthathela ingqalelo oko Itekhnoloji yeDatha eNkulu ayisiyiyo panacea ngokwayo. Ke ucaphula enye enkulu yetekhnoloji enje Isigculelo, unokongezwa ukuba:

«Ukuchonga ixabiso ledatha enkulu akuthethi nje ukuyihlalutya (esele iluncedo ngokwayo). Yinkqubo yonke yokufumanisa ukuba ifuna abahlalutyi, abasebenzisi beshishini kunye nabaphathi ukuba babuze imibuzo eyiyo, bachonge iipateni, benze izigqibo benolwazi kunye nokuqikelela isimilo. "

Izicelo ze-SL / CA zeDatha eNkulu

Phakathi kweSoftware yasimahla kunye nokuSebenzisa uMthombo ovulekileyo ekufuneka uchazelwe uphando, ukuvavanywa kunye nokuphunyezwa zezi:

Ezinxulumene

• Iapache Hadoop: Iqonga lomthombo ovulekileyo elenziwe yiHadoop Distributed File System (HDFS), iHadoop MapReduce, kunye neHadoop eqhelekileyo.
• I-Avro: Iprojekthi ye-Apache ebonelela ngeenkonzo ze-serialization.
• UCassandra: Kuhanjiswe isiseko sedatha esingesosobudlelwane esekwe kwimodeli yokugcina ye , Iphuhliswe kwiJava.
• Chukwa: Isoftware eyilelwe ukuqokelelwa kwesikali esikhulu kunye nohlalutyo lweelogo zemisitho.
• Ukuhamba: Isoftware enomsebenzi ophambili kukuhambisa idatha ukusuka komnye umthombo ukuya kwenye indawo.
• HBase: Isiseko sedatha esekwe kwikholam (isiseko sedatha esekwe kwikholamu) esisebenza kwi-HDFS.
• Isidleke: "Indawo yokugcina idatha" izibonelelo ezilungiselela ukulawulwa kwemithamo emikhulu yedatha egcinwe kwimeko yokusasazwa.
• Jaql: Umsebenzi kunye nolwimi oluchazayo oluvumela ukuxhaphazwa kwedatha kwifomathi yeJSON eyilelwe ukuqhubekeka nomthamo omkhulu wolwazi.
• ULucene: Isoftware ebonelela ngamathala eencwadi kwisalathiso kunye nokuphanda kwisicatshulwa.
• oozie: Iprojekthi yemithombo evulekileyo eyenza ukuba ukuhamba komsebenzi kulungelelaniswe phakathi kweenkqubo nganye.
• Ihagu: Isoftware evumela abasebenzisi beHadoop ukuba bagxile ngakumbi ekuhlalutyeni zonke iiseti zedatha kwaye bachithe ixesha elincinci ekwakheni iMephu yokunciphisa iinkqubo.
• Umgcini wezilwanyana: Izibonelelo ezisisiseko kunye neenkonzo ezinokusetyenziswa zizicelo ukuqinisekisa ukuba iinkqubo ezikwiklasta zilandisiwe okanye zangqanyaniswa.

Zimele

Abanye baziwa ngokulinganayo, kodwa bengadibani neqonga lomthombo ovulekileyo iHadoop zezi:

• I-Elasticsearch: Ukukhangela okusekwe kwisicatshulwa esigcweleyo kunye nenjini yohlalutyo.
• IMongoDB: Indawo yedatha yeNoSQL esekwe kwimodeli yedatha yoxwebhu.
• UCassandra: Iprojekthi yomthombo ovulekileyo we-Apache eyilelwe ukulawulwa kwedatha yeNoSQL.
• I-CouchDB: Umthombo ovulekileyo wedatha yeNoSQL esekwe kwimigangatho eqhelekileyo yokufikeleleka ngokulula kunye nokuhambelana kwewebhu ezahlukeneyo.
• Ilanga: Injini yokukhangela evulekileyo esekwe kwilayibrari yeprojekthi yeLucene.
  Ezinye izixhobo ze-RDBMS: Iqela lam le-MySQL kunye neVoltDB.

Isiphelo

Ixesha lethu langoku (nangoko ngoku lilandelayo) litywiliselwe okanye lurhaxwa kwinani eliphezulu kunye nelikhulayo ledatha, enokuthini xa iyonke, kunabantu ngabanye. Ke ngoko, ukusetyenziswa kwetekhnoloji yeDatha eNkulu ngoku nakwixesha elizayo eliza kunceda uluntu, bonke ubuntu, ukufumanisa ubuninzi bezinto (iziganeko okanye izinto ezenziweyo), ezinokuthatha iminyaka emininzi ukuzifumanisa. , Ngaphandle kokusebenzisa le.

Ukususela Idatha enkulu kunye nezixhobo zayo zibonelela ngesantya esaneleyo sohlalutyo Hlahlela isiphumo esifunyenwe ngokukhawuleza kwaye uphinde usenze kwakhona amaxesha amaninzi njengoko kufuneka, ngexesha elifutshane, ukufumana elona xabiso lokwenyani okanye lisondeleyo uzama ukufikelela kulo. Ukuba ufumene umxholo weDatha eNkulu unomdla, ungasandisa isihloko kancinci ngokufunda le Ngxelo BBVA.