Kuronga Mapaipi Makuru-Makuru eLLM ane DSPy uye LangChain kune Akavandudzwa DevOps

Kuronga Mapaipi Makuru-Makuru eLLM ane DSPy uye LangChain kune Akavandudzwa DevOps

Introduction

Kuuya kweMakuru Mutauro Models (LLMs) kwakashandura munda we Natural Language Processing (NLP) uye Artificial Intelligence (AI). Nekudaro, sezvo kuomarara uye kuwanda kweLLM-based masisitimu kunowedzera, ndizvo zvinoitawo kudiwa kwesimba rakasimba uye rinoshanda orchestration uye otomatiki maturusi. Muchinyorwa chino, tichaongorora munyika yeDSPy neLangChain, miviri miviri ine simba inogonesa kuvakwa kwesimba uye scalable software masisitimu muzera reAI.

Dambudziko reMapaipi Makuru eLLM Mapaipi

Mapaipi makuru eLLM anosanganisira akawanda maficha, kusanganisira data preprocessing, modhi yekudzidziswa, uye inference. Aya mapaipi anogona kuve akaoma, ane zvakawanda zvinotsamira uye kudyidzana pakati pezvikamu zvakasiyana. Sezvo huwandu hwemapaipi aya hunowedzera, ndizvo zvinoitawo njodzi yekukanganisa, kunonoka, uye mabhodhoro ezvishandiso.

Pinda DSPy uye LangChain

DSPy neLangChain mapaipi maviri anovavarira kurerutsa orchestration uye automation yemapaipi makuru eLLM. DSPy iPython-based framework inopa nzira yakapfava uye inoshanda yekuvaka nekugadzirisa yakaoma data mapaipi, nepo LangChain iri hurongwa hunogonesa kuvakwa kweanokatyamadza uye akasimba maitiro eAI.

DSPy: Iyo Python-Yakavakirwa Kusimukira yeData Pipelines

DSPy iPython-based framework inopa nzira yakapfava uye inoshanda yekuvaka nekugadzirisa yakaoma data mapaipi. NeDSPy, vagadziri vanogona kugadzira mapaipi edata ane scalable, akasimba, uye ari nyore kuchengetedza.

Heino muenzaniso wepombi yakapusa yedata yakavakwa neDSPy:

import dSpy se dsp


data_source = dsp.Source(
    zita = "my_data_source",
    mhando = "csv",
    nzira="/path/to/data.csv"
)

# Tsanangura data processor
data_processor = dsp.Processor(
    zita = "my_data_processor",
    mhando = "pandas",
    func=lambda x: x.dropna()
)

# Tsanangura sink yedata
data_sink = dsp.Sink(
    zita = "my_data_sink",
    mhando = "csv",
    nzira="/path/to/output.csv"
)

# Gadzira pombi yedata
pombi = dsp.Pipeline(
    zita = "rangu_pipeline",
    masosi=[data_source],
    processors=[data_processor],
    kunyura=[data_sink]
)

# Mhanya pombi
pipeline.run()

Mumuenzaniso uyu, tinotsanangura kwainobva data, data processor, uye sink yedata, tobva tagadzira pombi yedata inobatanidza zvikamu izvi. Iyo pombi inobva yaitwa uchishandisa run() nzira.

**LangChain: Chimiro cheScalable uye Resilient AI Systems **

LangChain chimiro chinogonesa kuvakwa kweanokatyamadza uye akasimba eAI masisitimu. NeLangChain, vagadziri vanogona kugadzira AI masisitimu ari nyore kuchengetedza, anotsemuka, uye anoshingirira mukutadza.

Heino muenzaniso weiyo yakapusa AI sisitimu yakavakwa neLangChain:

import langchain selc

# Tsanangura modhi
muenzaniso = lc.Model(
    zita = "my_model",
    mhando = "transformer",
    nzira="/nzira/ku/muenzaniso"
)

# Tsanangura chirevo
kukurumidza = lc.Prompt(
    zita = "zvangu_kukurumidza",
    text="Uyu muenzaniso wekukurumidza"
)

# Gadzira iyo AI system
ai_system = lc.AISystem(
    zita = "my_ai_system",
    muenzaniso=muenzaniso,
    kukurumidza=kukurumidza
)

# Mhanya iyo AI system
ai_system.run()

Mumuenzaniso uyu, tinotsanangura modhi, kukurumidza, uye tobva tagadzira AI system inobatanidza izvi zvikamu. Iyo AI sisitimu inozomhanya uchishandisa iyo run () nzira.

Kuronga Mapaipi Makuru-Makuru eLLM ane DSPy uye LangChain

Kuronga mapaipi makuru eLLM ane DSPy neLangChain, tinogona kushandisa nzira inotevera:

Tsanangura pombi yedata uchishandisa DSPy inobatanidza zvinyorwa zvakawanda, mapurosesa, uye sink.
Tsanangura chirongwa cheAI uchishandisa LangChain inobatanidza mhando dzakawanda, zvinokurudzira, uye zvinobuda.
Shandisa DSPy kutarisira pombi yedata uye LangChain kutarisira AI system.
Shandisa purogiramu yakadai seApache Airflow kana Kubernetes kuronga kushandiswa kwepombi yedata uye AI system.

Heino muenzaniso weyakareruka orchestration script inoshandisa DSPy uye LangChain:

import dSpy se dsp
import langchain selc
import apache_airflow as af

# Tsanangura pombi yedata
data_pipeline = dsp.Pipeline(
    zita = "my_data_pipeline",
    zvinyorwa=[dsp.Source(zita="my_data_source", type="csv", path="/path/to/data.csv")],
    processors=[dsp.Processor(zita="my_data_processor", type="pandas", func=lambda x: x.dropna())],
    sinks=[dsp.Sink(zita="my_data_sink", type="csv", path="/path/to/output.csv")]
)

# Tsanangura iyo AI system
ai_system = lc.AISystem(
    zita = "my_ai_system",
    model=lc.Model(zita = "my_model", type="transformer", nzira="/nzira/ku/model"),
    prompt=lc.Prompt(zita = "my_prompt", chinyorwa="Uyu muenzaniso wekukurumidza")
)

# Ronga kuitiswa kwepombi yedata uye AI system
af.schedule_task(
    task_id="my_task",
    dag_id="my_dag",
    pipeline=data_pipeline,
    system=ai_system
)

Mumuenzaniso uyu, tinotsanangura pombi yedata uye AI sisitimu inoshandisa DSPy neLangChain, uye tozoronga kuitwa kwezvikamu izvi uchishandisa Apache Airflow.

Pamusoro 5-10 Inonyanya Kushandiswa Mirairo/Zvidimbu

Heano epamusoro 5-10 anonyanya kushandiswa mirairo/zvimedu zveDSPy uye LangChain:

dsp.Pipeline(zita="rangu_pombi", matsime=[mabviro], processors=[processor], sink=[sink])
lc.Model(zita="my_model", type="transformer", nzira="/nzira/ku/muenzanisi")
lc.Prompt(zita = "zvangu_kukurumidza", chinyorwa="Uyu muenzaniso wekukurumidza")
dsp.Source(zita="my_data_source", type="csv", path="/path/to/data.csv")
dsp.Processor(zita="my_data_processor", type="pandas", func=lambda x: x.dropna())
dsp.Sink(zita="my_data_sink", type="csv", path="/path/to/output.csv")
lc.AISystem(zita="my_ai_system", modhi=muenzanisi, kukurumidza=kukurumidza)
af.schedule_task(task_id="my_task", dag_id="my_dag", pipeline=pipeline, system=system)
dsp.run_pipeline(pipeline)
lc.run_system(system)

Kuona

Heino ratidziro yeDSPy neLangChain masisitimu uchishandisa Chart.js:

{
  "type": "bha",
  "data": {
    "labels": ["DSPy", "LangChain"],
    "datasets": [
      {
        "label": "Zvinhu",
        "data": [10, 15],
        "backgroundColor": [
          "rgba(255, 99, 132, 0.2)",
          "rgba(54, 162, 235, 0.2)"
        ],
        "borderColor": [
          "rgba(255, 99, 132, 1)",
          "rgba(54, 162, 235, 1)"
        ],
        "borderWidth": 1
      }
    ]
  },
  "sarudzo": {}
}

Iyi yekuona inoratidza huwandu hwezvinhu muDSPy uye LangChain.

Mhedziso

DSPy neLangChain zvigadziriso zviviri zvine simba zvinogonesa kuvakwa kweakasimba uye anogona scalable software masisitimu muzera reAI. Nekushandisa masisitimu aya, vagadziri vanogona kugadzira mapaipi makuru eLLM ari nyore kuchengetedza, anokwirisa, uye anoshingirira mukutadza. Nerubatsiro rwe orchestration uye otomatiki maturusi akadai seApache Airflow uye Kubernetes, vanogadzira vanogona kuronga kuitiswa kweaya mapaipi nemasisitimu, zvichiita kuti zvive nyore kubata yakaoma AI masisitimu.

Basa Remangwana

Basa remangwana rinosanganisira:

Kugadzira mamwe maitiro eDSPy neLangChain.
Kuvandudza kushanda uye scalability yeDSPy neLangChain.
Kugadzira zvimwe zvekushandisa uye otomatiki zvishandiso zveDSPy neLangChain.
Kugadzira mamwe maratidziro uye dashboards eDSPy neLangChain.

Mareferensi

DSPy zvinyorwa: https://dspy.readthedocs.io/en/latest/
LangChain zvinyorwa: https://langchain.readthedocs.io/en/latest/
Apache Airflow zvinyorwa: https://airflow.apache.org/docs/apache-airflow/stable/index.html
Kubernetes zvinyorwa: https://kubernetes.io/docs/home/

Kuronga Mapaipi Makuru-Makuru eLLM ane DSPy uye LangChain kune Akavandudzwa DevOps

Have Questions?