Fungura moderi ya LLM na tokenizer | B4N1

Fungura moderi ya LLM na tokenizer

3 months ago · b4n1 Read time: 4 min

Gushyira mubikorwa Iperereza ryikora ryihuse hamwe nabakozi ba DevOps na LLMs

Intangiriro

Muri iki gihe cyihuta cyane cya digitale, amakipe ya DevOps ahura ningorabahizi: kumenya no gukemura ibyabaye vuba bishoboka. Uburyo bwiperereza bwintoki burashobora gutwara igihe kandi bikunda kwibeshya kubantu, biganisha kumasaha menshi kandi bikagabanuka kubakiriya. Kugira ngo iki kibazo gikemuke, turimo gukora ubushakashatsi ku ihuriro ry’imikorere ya DevOps, iperereza ryakozwe na AI, n’uruhare rwa Moderi nini y’indimi (LLMs) mu kuzamura ibikoresho bya DevOps nka AWS DevOps Agent na Copado Agentia.

Imiterere yiperereza ryibyabaye

Iperereza ryibyabaye nikintu gikomeye cya DevOps, kirimo kumenya, gusesengura, no gukemura ibibazo bireba sisitemu ya software. Uburyo gakondo bushingira kubikorwa byintoki, nko gusesengura ibiti, gukurikirana amakosa, no gutumanaho nabafatanyabikorwa. Mugihe ubu buryo bukora neza, burashobora gutinda kandi bukunze kwibeshya, biganisha kuri:

1 .. 2. ** Ikosa ryabantu **: Inzira zintoki zirashobora kwibasirwa namakosa yabantu, bishobora gutera kwisuzumisha nabi no gukemura ibyabaye. 3. ** Ibibazo byapimwe **: Mugihe ingano nuburemere bwa sisitemu bigenda byiyongera, uburyo bwiperereza bwintoki buragenda bugora gupima.

Uruhare rwabakozi ba DevOps mugushakisha ibyabaye

Abakozi ba DevOps, nka AWS DevOps Agent na Copado Agentia, bafite uruhare runini mugutangiza iperereza ryibyabaye. Izi ntumwa zirashobora:

  1. ** Kusanya no gusesengura ibiti **: Abakozi ba DevOps barashobora gukusanya no gusesengura ibiti biva ahantu hatandukanye, bitanga ubushishozi bwingenzi kubitera ibyabaye.
  2. ** Gutezimbere itumanaho **: Abakozi barashobora koroshya itumanaho hagati yamakipe, abafatanyabikorwa, na sisitemu, bakemeza ko amakuru yibyabaye asangiwe neza.

Uruhare rwa LLM mu iperereza ryibyabaye

Ururimi runini rwindimi (LLMs) rwahinduye urwego rwo gutunganya ururimi karemano (NLP), rutuma imashini zumva kandi zikabyara inyandiko zimeze nkabantu. Mu rwego rwo gukora iperereza ku byabaye, LLMs irashobora:

  1. ** Gusesengura raporo zibyabaye **: LLMs irashobora gusesengura raporo zibyabaye, ikamenya amakuru yingenzi nuburyo bishobora kwerekana intandaro yibyabaye.
  2. ** Gukora incamake yibyabaye **: LLMs irashobora gutanga incamake yincamake ya raporo yibyabaye, byorohereza itumanaho no gufata ibyemezo. D.

Gushyira mubikorwa Iperereza ryikora ryihuse hamwe nabakozi ba DevOps na LLMs

Kugirango dushyire mubikorwa iperereza ryikora hamwe nabakozi ba DevOps na LLMs, tuzakurikiza intambwe ku yindi:

  1. ** Huza abakozi ba DevOps **: Huza abakozi ba DevOps, nka AWS DevOps Agent na Copado Agentia, hamwe na sisitemu yo gucunga ibyabaye.
  2. ** Kusanya no gusesengura ibiti **: Hindura abakozi ba DevOps gukusanya no gusesengura ibiti biva ahantu hatandukanye, bitanga ubushishozi bwimpamvu zitera.
  3. ** Hugura icyitegererezo cya LLM **: Hugura icyitegererezo cya LLM kuri dataset ya raporo yibyabaye, ibiti, nandi makuru afatika.
  4. ** Tanga igisubizo **: Kohereza igisubizo cyiperereza ryakozwe ryihuse, urebe ko ari kinini, cyizewe, kandi gifite umutekano.

Kode yintangarugero: Guhuza LLM hamwe nabakozi ba DevOps

Dore urugero kode yerekana muri Python, yerekana uburyo bwo guhuza moderi ya LLM numukozi wa DevOps: `` python gutumiza mu mahanga kuva muri transformateur itumiza AutoModelForSequenceClassification, AutoTokenizer kuva aws_devops_agent yatumije abakozi

icyitegererezo = AutoModelForSequenceClassification. Kuva_yateguwe ("distilbert-shingiro-idafite ishingiro") tokenizer = AutoTokenizer. kuva_yateguwe ("distilbert-shingiro-idafite ishingiro")

Tangiza umukozi wa DevOps

umukozi = Intumwa ()

Sobanura imikorere yo gusesengura raporo zibyabaye ukoresheje moderi ya LLM

def gusesengura_impanuka_amakuru (raporo): # Tegura raporo y'ibyabaye inyongeramusaruro = tokenizer.encode_plus (raporo, ongeraho_idasanzwe_tokens = Nukuri, max_uburebure = 512, garuka_kwitonda_mask = Nukuri, garuka_tensors = 'pt')

# Gisesengura raporo y'ibyabaye ukoresheje icyitegererezo cya LLM
ibisubizo = icyitegererezo (inyongeramusaruro ['ibyinjijwe_id'], kwitondera_mask = inyongeramusaruro ['kwitondera_mask'])

# Kuramo ikirango cyahanuwe
byahanuwe_cyiciro = itara.argmax (ibisohoka.logits)

garuka byahanuwe

Sobanura imikorere yo gutanga ibyifuzo byo gukemura ibyabaye

def gutanga_ibyifuzo (byahanuwe_icyiciro): # Koresha ikirango cyahanuwe kugirango utange ibyifuzo byo gukemura ibyabaye niba byahanuwe_cyiciro == 0: garuka "Reba ibiti bya sisitemu ku makosa" elif yahanuye_cyiciro == 1: garuka "Kugenzura umuyoboro uhuza" ikindi: garuka "Menyesha itsinda ryiterambere kugirango bagufashe"

Huza moderi ya LLM n'umukozi wa DevOps

def ibyabaye_ubushakashatsi (): # Kusanya no gusesengura ibiti bivuye muri agent wa DevOps ibiti = umukozi.ibikoresho ()

# Gisesengura ibiti ukoresheje icyitegererezo cya LLM
byahanuwe_ibyiciro = gusesengura_impanuka_yerekana (ibiti)

# Tanga ibyifuzo byo gukemura ibyabaye
ibyifuzo = gutanga_ibyifuzo (byahanuwe_icyiciro)

garuka ibyifuzo

Tanga igisubizo cyiperereza ryibyabaye

ibyabaye_ubushakashatsi () `` ` Hejuru 5-10 Amabwiriza Yakoreshejwe Byinshi / Udukoryo

Hano haribintu 5-10 byambere bikoreshwa cyane / ibisobanuro kuri ubu buhanga:

  1. "Aws devops agent get-logs": Kusanya ibiti bivuye mubakozi ba DevOps.
  2. `abahindura.
  3. tokenizer.encode_plus (): Tegura raporo yibyabaye.
  4. `icyitegererezo (inyongeramusaruro ['ibyinjijwe_ibikoresho'], kwitondera_mask = inyongeramusaruro
  5. torch.argmax (ibisohoka.logits): Kuramo ikirango cyahanuwe.
  6. gutanga_ibyifuzo (byahanuwe_cyiciro): Tanga ibyifuzo byo gukemura ibyabaye.
  7. agent.get_logs (): Kusanya ibiti bivuye mubakozi ba DevOps.
  8. gusesengura_impanuka_amakuru (raporo): Gusesengura raporo yibyabaye ukoresheje icyitegererezo cya LLM.
  9. ibyabaye_ubushakashatsi (): Kohereza igisubizo cyiperereza ryibyabaye.
  10. torch.save (moderi.state_dict (), 'icyitegererezo.pth'): Bika icyitegererezo cya LLM.

Kubona amashusho

Hano hari amashusho yuburyo bwihuse bwo gukora iperereza ukoresheje Chart.js: json { "ubwoko": "akabari", "amakuru": { "ibirango": ["Iperereza ry'intoki", "Iperereza ryikora"], "imibare": [ { "label": "Igihe cyo Gukemura", "amakuru": [10, 2], "backgroundColor": ["rgba (255, 99, 132, 0.2)", "rgba (54, 162, 235, 0.2)"], "umupaka w'amabara": ["rgba (255, 99, 132, 1)", "rgba (54, 162, 235, 1)"], "Ubugari bwumupaka": 1 } ] }, "amahitamo": { "umutwe": { "kwerekana": ukuri, "inyandiko": "Igihe cyo gukemura ibyabaye" }, "umunzani": { "yAxes": [ { "amatiku": { "gutangiraAtZero": ukuri } } ] } } } ` Iyi visualisation igereranya igihe cyo gukemura ibyabaye ukoresheje uburyo bwiperereza ryintoki nuburyo bwiperereza bwikora ukoresheje abakozi ba DevOps na LLMs.

Umwanzuro

Gushyira mubikorwa iperereza ryikora hamwe nabakozi ba DevOps hamwe na LLMs birashobora kunoza cyane imikorere nubushobozi bwo gukemura ibyabaye. Muguhuza abakozi ba DevOps hamwe na LLMs, amashyirahamwe arashobora guhita akora isesengura ryibiti, agatanga ibitekerezo-byerekeranye nibisobanuro byo gukemura ibyabaye, kandi bikagabanya igihe cyo gukemura ibibazo. Twakoze ubushakashatsi ku masangano yimikorere ya DevOps, iperereza ryakozwe na AI, nuruhare rwa LLM mugutezimbere ibikoresho bya DevOps nka AWS DevOps Agent na Copado Agentia. Mugukurikiza inzira-ku-ntambwe ivugwa muri iyi ngingo, amashyirahamwe arashobora gukoresha igisubizo cyipimishije, cyizewe, kandi gifite umutekano.


Have Questions?

Let's Talk

Contact me on LinkedIn