Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Trying to use streaming with tools with Ollama via OpenAI is broken #2289

Closed
edeandrea opened this issue Dec 17, 2024 · 3 comments · Fixed by #1989
Closed

[BUG] Trying to use streaming with tools with Ollama via OpenAI is broken #2289

edeandrea opened this issue Dec 17, 2024 · 3 comments · Fixed by #1989
Labels
bug Something isn't working Streaming Tools

Comments

@edeandrea
Copy link
Contributor

edeandrea commented Dec 17, 2024

This was reported as quarkiverse/quarkus-langchain4j#1164 but was determined to be an issue in the core LangChain4j. I'll copy/paste the issue content from there:

When trying to use Tools with Ollama in streaming mode via the OpenAI API it doesn't seem to work right. It works fine if I use the OpenAI extension to talk to ChatGPT, or the Ollama extension to talk to Ollama. It only seems to be a problem if using the OpenAI endpoint for Ollama.

To reproduce:

  1. Clone the https://github.com/edeandrea/parasol-insurance/tree/rag-ollama-openai-api (the rag-ollama-openai-api branch from https://github.com/edeandrea/parasol-insurance)
  2. cd app
  3. Start Ollama
  4. ollama pull llama3.2 && ollama pull snowflake-arctic-embed
  5. ./mvnw clean quarkus:dev
  6. Once dev mode is up, hit the w key to open the app
  7. In the claims list page, click the 1st claim (Claim Number CLM195501)
  8. On the claim details page, click the chat bot button on the bottom right corner
  9. In the chat query, enter Update claim status to denied
  10. Notice the error in the logs:
2024-12-16 12:30:30,472 INFO  [io.qua.lan.ope.com.OpenAiRestApi$OpenAiClientLogger] (vert.x-eventloop-thread-3) Request:
- method: POST
- url: http://localhost:11434/v1/embeddings
- headers: [Accept: application/json], [Authorization: Be...me], [Content-Type: application/json], [User-Agent: langchain4j-openai], [content-length: 1168]
- body: {
  "model" : "snowflake-arctic-embed",
  "input" : [ "    Claim ID: 1\n\n    Policy Inception Date: 1954-09-30\n\n    Claim Summary:\n    \n    On January 2nd, 1955, at around 3:30 PM, a car accident occurred at the intersection of Colima Road and Azusa Avenue in Hill Vallet. The involved parties were Marty McFly, driving a silver Delorean DMC-12 (OUTA-TIME), and Biff Tanner in a blue Type 2 Volkswagen Bus (BIF-RULZ).\n\n    Marty was heading south on Colima Road when Biff failed to stop at the red traffic signal on Asuza Avenue, causing a collision with Marty's vehicle. Both drivers exchanged information and took photos of the accident scene, which included damages to the front driver and passenger side of Marty's Delorean DMC-12 and the front driver's side of Biff's Volkswagen Bus. No injuries were reported.\n\n    Marty has attached necessary documents, such as photos, a police report, and an estimate for repair costs, to his email. He requests prompt attention to the claim and is available at (916) 555-4385 or marty.mcfly@email.com for any additional information or documentation needed.\n    \n\n    Question: Update claim status to denied\n" ]
}

2024-12-16 12:30:30,638 INFO  [io.qua.lan.ope.com.OpenAiRestApi$OpenAiClientLogger] (vert.x-eventloop-thread-3) Response:
- status code: 200
- headers: [Content-Type: application/json], [Date: Mon, 16 Dec 2024 17:30:30 GMT], [Transfer-Encoding: chunked]
- body: {"object":"list","data":[{"object":"embedding","embedding":[0.051931795,-0.02034702,-0.026558196,0.00050836895,0.010104616,0.018725052,0.01966646,0.014132896,-0.02420254,-0.007781001,-0.021436322,0.045459103,-0.042137217,0.0005802696,0.04656498,0.033859845,-0.0030202188,0.0007024488,-0.012138537,0.0070828684,0.028452003,-0.023737537,0.03920396,-0.030042518,0.028599115,-0.0020970297,-0.039480466,-0.010599784,-0.0032752487,-0.02346951,-0.031322703,-0.0024191546,0.025104178,-0.031132251,0.0039812597,0.04494699,0.028225493,-0.0130562335,-0.021262715,-0.07110066,0.011589404,0.017965283,0.019505108,0.043088723,-0.029474068,-0.051794503,0.06390038,0.035999753,-0.021681378,-0.048631627,0.018944006,-0.08328185,-0.0064189546,-0.0008961293,0.001045079,-0.049308535,-0.028374929,0.03570221,0.021617897,0.002131446,0.014445037,-0.0018198548,-0.041406438,-0.028520135,0.0014781143,0.012868444,-0.004367552,0.041141618,-0.013034408,-0.047680493,0.026334055,-0.00752851,-0.0063327355,0.01121876,0.09142511,-0.06924804,0.034741428,-0.04699025,0.010148355,0.01177471,0.028833495,-0.0114557175,0.022345023,-0.04050431,0.029326696,-0.00009956705,0.035981983,0.04783336,0.024965785,-0.0022750327,-0.0390766,0.000047302034,-0.013255039,-0.0014829882,-0.012854186,-0.015006039,-0.040952023,0.040163983,0.006395965,0.0042811693,-0.0052510826,-0.012133593,-0.0120742945,0.04256371,-0.035018425,0.038852137,-0.02769264,-0.03676663,-0.008812307,-0.012543874,-0.025429297,-0.0030992546,0.00028597293,-0.00045889526,0.00030044268,0.03487327,0.004167487,0.0036690603,0.048414066,-0.018980343,0.04106293,-0.03554645,-0.0063533885,-0.036866803,0.052073173,0.067129955,-0.04904762,-0.0029488914,-0.051635843,0.019754201,-0.027429216,-0.03392541,0.038766466,-0.02962561,-0.05605364,0.037883807,-0.045427173,-0.044253226,0.015683575,0.040710293,-0.054342818,-0.05889429,-0.0127986325,-0.010523816,0.032328036,0.01968902,0.039801307,0.020128222,0.010626442,-0.056038674,-0.018227436,-0.01897297,0.010014654,-0.023497762,0.0014940929,-0.06584621,-0.00609589,-0.0039708405,-0.048092913,-0.026879467,0.084838286,0.007322367,0.0020586585,-0.020346146,-0.032146003,-0.0021742191,0.052095942,0.015225747,0.041047115,-0.01706927,-0.0044795275,0.046106204,-0.0055230577,0.021889621,0.027876714,-0.025241533,0.012097388,0.05117452,0.006620187,-0.03035234,0.021581648,-0.02363841,-0.007036968,0.03027711,-0.023547282,0.008250964,-0.02296185,0.025395833,-0.011424631,0.055308767,-0.02862333,0.038770117,-0.02679813,0.012391687,-0.043218203,-0.008297593,-0.06348765,0.05035021,0.007051335,0.013865221,0.043054912,-0.010386098,0.05841244,0.0007113601,0.004832827,0.016538002,0.010746147,0.0075005367,-0.06641261,-0.06964288,0.048441157,0.023249248,-0.006778648,0.016213607,-0.0023551984,0.06098773,0.043543328,-0.0032884248,0.031352174,0.036176044,0.007927931,-0.020372616,0.055584267,0.05093917,0.0028953922,-0.007187078,-0.077051826,-0.03242927,-0.009709336,-0.022349944,0.003119497,-0.019169383,-0.017247032,-0.043140866,0.055257313,-0.03150914,-0.01802525,0.00889391,-0.00794501,0.044323027,0.049971804,-0.011902749,-0.0063349283,0.041635506,0.0014675925,0.016931707,-0.0645525,-0.047553502,-0.0015944429,0.047738336,-0.012418087,0.0005887973,0.048492126,-0.03527503,-0.014565046,0.02683408,0.013133723,-0.09153304,0.033625837,0.01934981,-0.0047992337,0.020364998,0.033898775,-0.0421141,0.0067186058,0.016231714,0.009713957,0.01967671,0.061220776,0.008998302,0.019669093,-0.029900014,-0.019740384,0.0060445773,0.013088268,-0.038355537,-0.0013782535,0.008978301,0.010503949,0.026242001,-0.032497562,0.020915717,-0.018446134,-0.024556262,0.009844473,0.00850268,0.0029606498,-0.009906723,0.034954272,0.04817554,-0.039350346,-0.030310236,-0.009657999,-0.01812264,-0.010562317,0.055074427,-0.020667702,-0.018409403,-0.058746707,0.018544521,0.0020697962,0.016337797,0.024903303,-0.03335062,-0.04145707,-0.0371577,-0.034135025,-0.0006172894,0.00007170714,0.029747106,-0.049574926,0.0056340788,-0.034737956,-0.014256746,-0.08596452,-0.02821785,-0.04258316,-0.009687731,-0.017132388,-0.011849246,-0.015866566,0.009624052,0.0017399994,0.009152291,-0.013551168,-0.0019566703,-0.029426664,-0.032338757,0.009032642,-0.014970659,-0.018287646,0.034956705,0.03127019,-0.00793427,-0.012431051,-0.013663917,0.007978364,-0.00047537094,-0.020531872,0.038308028,0.06812182,0.013470123,0.013090675,0.0031569952,-0.012873647,0.01013838,0.011738239,0.018259987,0.046681017,0.02699981,0.0017427722,-0.01638603,0.0012878748,-0.01847961,-0.033712678,-0.005971678,0.026710307,-0.059111085,-0.018426644,0.040561482,-0.01330151,0.012083137,-0.017907832,-0.002831174,-0.020356188,0.0015017354,-0.008254642,-0.007986318,0.014124353,0.03510749,0.027879415,0.020743415,-0.025277108,0.006215059,-0.043507725,-0.035352066,0.030724887,-0.03488176,-0.02503465,-0.038111467,-0.008129879,0.012680543,0.005629714,-0.030001653,-0.059074104,-0.00083021255,0.01061359,-0.02339689,-0.009804693,-0.008512907,0.060777687,0.009069269,0.063368745,-0.024350699,-0.005908278,-0.005950892,0.02429447,-0.041832585,-0.06183725,-0.015223698,-0.0036805652,0.073426776,0.006050309,-0.036783993,-0.0038472344,0.0007893965,0.008607965,-0.014330405,-0.06727822,-0.043499123,0.03620355,-0.033218388,-0.018988883,0.021018824,0.01207771,0.02960999,-0.0037041297,-0.0025444923,0.0059225773,-0.021842837,0.04148992,-0.0245756,-0.019086488,-0.042076252,-0.025681555,0.031102648,-0.0022974687,0.018829312,0.005347805,0.031188041,0.014918501,0.0029079206,-0.002172845,0.0033112224,0.0069908765,0.037058387,-0.038806334,0.00174555,0.052802272,0.00877139,-0.027891874,0.020342369,-0.0010967838,-0.02438576,0.0016055453,-0.036264457,-0.054650575,0.05941556,0.0026594787,0.019159969,0.041671623,-0.00029286186,0.01992144,0.019459149,-0.029387971,-0.06454728,-0.023852006,0.036596257,0.044435248,-0.037368845,-0.028511459,-0.0049129245,-0.039350834,0.02059841,0.0120839495,0.008199784,0.035526358,-0.07683122,-0.06617066,-0.019686393,0.014590975,0.007388091,0.05680725,0.043391723,-0.036233153,-0.013851317,0.023160545,0.04317247,0.0103298705,-0.006315762,-0.034253377,0.00012070916,0.050787713,-0.040505975,0.01419512,-0.067052916,-0.041907698,0.034815505,0.06696849,0.056659415,0.06968395,0.02532013,0.047050066,0.018687505,0.03204649,0.02973243,0.017357977,0.0015282497,0.026633913,0.028596682,-0.0105165215,0.015007502,0.0035600893,0.03346389,-0.033998605,0.052162133,0.014961762,0.029114237,-0.01671867,-0.0007021343,0.023779381,-0.0019470641,-0.016348137,-0.02354427,0.01639167,-0.011119919,0.032109678,-0.008749375,-0.01435958,-0.0077493824,-0.0031517276,0.010898148,0.027753308,0.03277233,-0.052892126,-0.032437243,-0.004940298,-0.0017554123,-0.052395787,-0.019551625,0.0011363489,-0.008196913,-0.05778825,-0.000045438854,0.03370003,-0.01270665,0.0688374,-0.017260978,0.007991008,-0.031271666,-0.050928053,0.04344978,-0.01574733,0.018911546,0.021529086,0.012924509,0.000036838064,0.0050603426,-0.024613898,0.04500872,0.0043871617,0.0007568486,-0.020267418,-0.0009630664,0.004461692,-0.019222494,0.074243866,-0.01570982,-0.0018287374,-0.002991065,-0.018593717,-0.0037432506,0.009523753,0.001260094,0.04275937,0.049599275,-0.023975212,-0.034078352,-0.013791663,-0.011446572,-0.012210785,0.04287541,0.006745985,-0.032745987,-0.0011938284,0.054858625,0.040344395,0.0005363699,-0.0020523272,-0.0020874674,-0.022528568,-0.029616928,-0.0022039437,0.04330623,-0.031231502,-0.020203346,0.042609166,-0.016875297,0.0057349983,-0.05570488,-0.013184141,0.07871215,0.020173475,-0.012312273,0.045899447,0.006984953,-0.019274224,-0.0752313,-0.04041196,-0.0046295556,-0.03292471,-0.000035899237,0.041766457,-0.0003711141,0.0018228336,0.004426299,-0.021262022,0.061582316,-0.028943447,0.005870689,0.022012256,-0.0154758105,-0.014230208,-0.03356533,-0.022507062,0.035240617,0.014584642,0.03159748,0.022681694,-0.                                                                                                                                               01176882,-0.053956382,-0.03196748,0.026001364,0.0066515775,0.086453766,-0.0008501544,0.015080363,-0.0080821635,0.06151388,0.022731585,-0.0033829813,-0.016226487,0.016992655,0.012511048,-0.037473038,-0.023350053,-0.021108622,-0.010281363,0.03344744,0.013995421,0.03266687,0.072873734,-0.05463405,0.04979493,-0.014571098,-0.026067778,-0.022629969,0.010525153,-0.03893194,0.018862903,-0.021003416,0.06302875,-0.010185048,0.028593527,-0.005273706,-0.029586254,0.049048226,-0.022761444,-0.020388702,0.012239825,-0.0012336011,-0.0087627,-0.0037817585,0.01558477,-0.0032572707,0.13026372,0.04644816,-0.013943706,0.025220305,-0.047650494,0.05988558,-0.0069678067,0.00520945,0.033722233,0.055075336,-0.033418402,0.0731673,0.008310487,0.015801447,-0.04469987,0.027938155,-0.03566864,0.007301879,-0.0030115915,0.04827306,0.010401449,-0.03772671,0.007770854,0.025913667,-0.0042495592,-0.025273887,-0.009162389,-0.037172157,0.055654913,0.019021058,0.025324706,0.022265272,0.044682898,-0.0035963748,-0.0032611254,0.020117514,0.012163127,0.058320884,0.043215483,0.03102061,0.04426158,0.042613123,-0.006430045,-0.008584433,-0.03475602,0.018518358,-0.0123175625,0.059669018,-0.025497569,0.019108731,0.02869589,-0.051342584,0.06720214,-0.017038424,0.03455304,0.0017381657,0.03306871,-0.0058664177,0.036950443,0.0066603464,0.047212433,0.03047037,-0.010979447,0.030680735,-0.019431733,0.07477526,-0.0298297,0.096150875,0.0069834865,0.0062529207,-0.049069297,-0.011329122,0.023079915,0.030188423,0.009270117,-0.011901174,-0.03617495,0.034065083,-0.031800456,0.014659898,0.05063497,-0.029604109,0.047093503,-0.021763345,-0.029982891,-0.040199343,0.02223967,0.00043902674,-0.02621466,-0.025760742,-0.0005617702,0.04546059,-0.073916934,0.044096295,0.021029696,0.01566658,0.016204657,-0.013663736,-0.034873404,0.012292677,0.018687274,-0.0018178098,-0.021967744,0.014449967,0.014019527,0.01265758,0.07013359,-0.01907598,-0.03549252,-0.033728033,0.057539236,-0.027234746,0.002681768,-0.034867935,-0.020151727,0.028034993,0.00045261145,-0.012993105,0.0028326306,-0.033593703,-0.032240547,-0.005603399,0.013804566,0.019316047,0.059404615,-0.04287804,-0.011235002,-0.006295801,-0.045856312,-0.016091082,-0.009254973,-0.026902638,-0.005412072,-0.042010073,0.026465245,-0.040632643,-0.02203935,0.0149122225,-0.005939665,-0.020063912,-0.010323515,-0.003182312,-0.0060487287,-0.012649852,-0.0026798744,0.010059464,0.009567971,0.042513944,-0.0033655155,0.038396843,0.009884965,-0.0229289,-0.00026675352,-0.031779576,-0.008155306,0.009277366,0.05033351,0.013317088,0.0051822453,-0.07742422,-0.020824848,0.03113489,0.035879496,-0.017384803,0.026371911,-0.042036474,0.002020759,0.0019143333,0.012587888,-0.0043834182,0.016339513,-0.022565145,0.044902198,-0.00074123277,-0.04294496,-0.021520196,0.05614885,0.053641137,0.018772442,0.00063462864,0.009979474,-0.02315693,0.015668523,-0.002918376,-0.04542177,0.00036650093,-0.018193787,0.0029440369,-0.056138083,0.02074343,0.01865178,-0.020517446,0.04080733,0.006017338,-0.02192589,-0.030179022,-0.018567812,-0.050420135,-0.040512085,-0.030890252,0.03070048,-0.00033151935,0.07575551,0.019991513,0.00032171648,-0.008321291,-0.0338683,-0.056124005,0.036848024,-0.036674716,0.007996757,-0.0010411638,0.008382041,0.020364678,-0.013363147,0.0078018866,0.0033296933,0.027298363,-0.04786229,0.03163783,-0.021487601,0.013263743,0.021667408,0.0055645145,-0.009621346,-0.0029328358,-0.028112056,0.0071598925,0.01839052,-0.039669544,-0.033413596,0.0022446017,-0.012156612,0.023722518,-0.0098790275,-0.0328877,-0.014284377,-0.009042453,0.0034010184,-0.014517193,0.0058016637,0.03203535,-0.012101081,0.0040990915,0.00549322,0.043262944,0.010602458,0.01403057,0.015290817,-0.051710695,0.011995105,-0.03486301,0.014584558,-0.0055276603,-0.008176099,0.008719396,0.016337369,-0.0005166559,-0.031645294,0.04645941,0.053469285,-0.010584689,-0.027876008,0.0008710384,-0.01626717,-0.016103523,-0.002839306,-0.036901217,-0.0055812504,0.009995682,0.0530507,0.013502026,0.038063865,-0.03190211,-0.010379636,-0.055024043,0.016907256,0.03987505,-0.027537158,-0.043551546,-0.054334044,0.014710043,0.017099781,0.0036349092,0.015791351,0.024266727,-0.015664505,-0.014498972,-0.07083478,0.05766795,0.044493895,-0.041765723,0.030378118,-0.015268891,-0.0036668326,-0.085799664,0.025580091,0.02242234,0.070310436,0.0145874005,-0.042899277,-0.012330911,0.042385556,0.030886823,0.005596077,-0.029579042,-0.04856982,-0.008554978,-0.00043708298,-0.030364526,0.024286758,-0.01085181,0.007301938,-0.03463167,0.00072832935,0.025639508,-0.031213652,0.019692047,0.067772955,-0.013281161,0.0032380512,0.011064156,0.043444507,0.013783356,-0.007147444,0.015527564,0.008425218,0.052399464,0.024706405,-0.018904017,-0.012396661,-0.012973677,-0.01531046,-0.04311064,-0.012149753,0.03737271,-0.0046595866,-0.011961291,0.000105140396,-0.01259426,0.06093836,0.0078358445,-0.03217898,-0.050812338,-0.046601083,0.013346951,-0.047487866,0.006093036,-0.056887493,-0.032009806,-0.021170922,-0.04987386,0.039206877,0.02288925,-0.028373701,-0.012024-12-16 12:30:30,777 INFO  [io.qua.lan.ope.com.OpenAiRestApi$OpenAiClientLogger] (vert.x-eventloop-thread-3) Request:total_tokens":248}}
- method: POST
- url: http://localhost:11434/v1/chat/completions
- headers: [Accept: text/event-stream], [Authorization: Be...my], [Content-Type: application/json], [User-Agent: langchain4j-openai], [content-length: 5115]
- body: {
  "model" : "llama3.2",
  "messages" : [ {
    "role" : "system",
    "content" : "You are a helpful, respectful and honest assistant named \"Parasol Assistant\".\nYou will be given a claim summary, references to provide you with information, and a question. You must answer the question based as much as possible on this claim with the help of the references.\nAlways answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.\n\nIf a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.\n\nYou must answer in 4 sentences or less.\n\nDon't make up policy term limits by yourself\n"
  }, {
    "role" : "user",
    "content" : "    Claim ID: 1\n\n    Policy Inception Date: 1954-09-30\n\n    Claim Summary:\n    \n    On January 2nd, 1955, at around 3:30 PM, a car accident occurred at the intersection of Colima Road and Azusa Avenue in Hill Vallet. The involved parties were Marty McFly, driving a silver Delorean DMC-12 (OUTA-TIME), and Biff Tanner in a blue Type 2 Volkswagen Bus (BIF-RULZ).\n\n    Marty was heading south on Colima Road when Biff failed to stop at the red traffic signal on Asuza Avenue, causing a collision with Marty's vehicle. Both drivers exchanged information and took photos of the accident scene, which included damages to the front driver and passenger side of Marty's Delorean DMC-12 and the front driver's side of Biff's Volkswagen Bus. No injuries were reported.\n\n    Marty has attached necessary documents, such as photos, a police report, and an estimate for repair costs, to his email. He requests prompt attention to the claim and is available at (916) 555-4385 or marty.mcfly@email.com for any additional information or documentation needed.\n    \n\n    Question: Update claim status to denied\n\n\nAnswer using the following information:\n4.3.\n\nAn adjuster may be assigned to inspect the vehicle and assess damages.\n\n5. Premiums and Deductibles\n5.1. Premiums are calculated based on factors including driver history, vehicle type, and coverage\n\nselected.\n5.2. Deductibles apply to collision and comprehensive coverage as specified on the declarations\n\npage.\n\n6. Cancellation\n6.1. The policyholder may cancel at any time with pro-rata refund of premiums.\n6.2. The insurer reserves the right to cancel within 30 days notice, or immediately in cases of\n\nnon-payment or fraud.\n\n7. Definitions\n7.1. “Insured vehicle” refers to the automobile listed on the declarations page.\n\n7.2. “Accident” means a sudden, unexpected event resulting in damage or injury.\n\n8. Additional Provisions\n8.1. Coverage extends to other drivers listed on the policy.\n8.2. Rental car coverage may be included if specified on the declarations page.\n8.3. Roadside assistance may be available if specified on the declarations page.\n\n9. Contact Information\nFor claims or inquiries:\nPhone: 800-CAR-SAFE\nEmail: claims@parasol.com\n\nParasol Insurance Company\nPolicy Document\n\n1. Coverage\n1.1. This policy provides coverage for the insured vehicle as specified in the declarations page.\n1.2. Coverage types may include:\n\n1.2.1. Liability (Bodily Injury and Property Damage)\n1.2.2. Collision\n1.2.3. Comprehensive\n1.2.4. Personal Injury Protection (PIP)\n1.2.5. Uninsured / Underinsured Motorist\n\n1.3. Specific coverage limits and deductibles are listed on the declarations page.\n\n2. Policy Term\n2.1. The policy term is six months from the date of inception, unless otherwise specified.\n2.2. Renewal is not automatic and must be initiated by the policyholder.\n\n3. Exclusions\n3.1. Damage caused by intential acts or gross negligence is not covered.\n3.2. Normal wear and tear is excluded from coverage.\n3.3. Damage resulting from racing is excluded\n3.4. Damage resulting from using the vehicle for commercial purposes without proper endorsement\n\nis excluded.\n3.5. Driving under the influence of alcohol or drugs voids coverage for that incident.\n\n4. Claim Process\n4.1. Claims should be filed as soon as possible after an incident.\n4.2. Policyholders must provide all relevant information, including police reports when applicable.\n4.3. An adjuster may be assigned to inspect the vehicle and assess damages."
  } ],
  "temperature" : 0.3,
  "top_p" : 1.0,
  "stream" : true,
  "stream_options" : {
    "include_usage" : true
  },
  "presence_penalty" : 0.0,
  "frequency_penalty" : 0.0,
  "tools" : [ {
    "type" : "function",
    "function" : {
      "name" : "updateClaimStatus",
      "description" : "update claim status",
      "parameters" : {
        "type" : "object",
        "properties" : {
          "claimId" : {
            "type" : "integer"
          },
          "status" : {
            "type" : "string"
          }
        },
        "required" : [ "claimId", "status" ]
      }
    }
  } ]
}

2024-12-16 12:30:36,213 DEBUG [org.par.res.ClaimWebsocketChatBot] (vert.x-eventloop-thread-3) Got chat response: 
2024-12-16 12:30:36,218 DEBUG [org.par.res.ClaimWebsocketChatBot] (vert.x-eventloop-thread-3) Got chat response: 
2024-12-16 12:30:36,219 ERROR [io.qua.ver.cor.run.VertxCoreRecorder] (vert.x-eventloop-thread-3) Uncaught exception received by Vert.x: java.lang.IllegalArgumentException: response cannot be null
        at dev.langchain4j.internal.Exceptions.illegalArgument(Exceptions.java:19)
        at dev.langchain4j.internal.ValidationUtils.ensureNotNull(ValidationUtils.java:50)
        at dev.langchain4j.internal.ValidationUtils.ensureNotNull(ValidationUtils.java:37)
        at dev.langchain4j.model.chat.listener.ChatModelResponseContext.<init>(ChatModelResponseContext.java:25)
        at dev.langchain4j.model.openai.OpenAiStreamingChatModel.lambda$generate$3(OpenAiStreamingChatModel.java:229)
        at io.smallrye.mutiny.subscription.Subscribers$CallbackBasedSubscriber.onCompletion(Subscribers.java:109)
        at io.smallrye.mutiny.operators.multi.builders.BaseMultiEmitter.completion(BaseMultiEmitter.java:57)
        at io.smallrye.mutiny.operators.multi.builders.BufferItemMultiEmitter.drain(BufferItemMultiEmitter.java:110)
        at io.smallrye.mutiny.operators.multi.builders.BufferItemMultiEmitter.completion(BufferItemMultiEmitter.java:68)
        at io.smallrye.mutiny.operators.multi.builders.BaseMultiEmitter.complete(BaseMultiEmitter.java:48)
        at io.smallrye.mutiny.operators.multi.builders.BufferItemMultiEmitter.complete(BufferItemMultiEmitter.java:11)
        at io.smallrye.mutiny.operators.multi.builders.SerializedMultiEmitter.drainLoop(SerializedMultiEmitter.java:117)
        at io.smallrye.mutiny.operators.multi.builders.SerializedMultiEmitter.drain(SerializedMultiEmitter.java:92)
        at io.smallrye.mutiny.operators.multi.builders.SerializedMultiEmitter.onCompletion(SerializedMultiEmitter.java:87)
        at io.smallrye.mutiny.operators.multi.builders.SerializedMultiEmitter.complete(SerializedMultiEmitter.java:154)
        at org.jboss.resteasy.reactive.client.impl.MultiInvoker$MultiRequest.complete(MultiInvoker.java:87)
        at org.jboss.resteasy.reactive.client.impl.SseEventSourceImpl.notifyCompletion(SseEventSourceImpl.java:220)
        at org.jboss.resteasy.reactive.client.impl.SseEventSourceImpl.close(SseEventSourceImpl.java:203)
        at org.jboss.resteasy.reactive.client.impl.SseEventSourceImpl.lambda$registerOnClient$2(SseEventSourceImpl.java:152)
        at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:270)
        at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:252)
        at io.vertx.core.http.impl.HttpEventHandler.handleEnd(HttpEventHandler.java:76)
        at io.vertx.core.http.impl.HttpClientResponseImpl.handleEnd(HttpClientResponseImpl.java:250)
        at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$new$0(Http1xClientConnection.java:421)
        at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:279)
        at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:157)
        at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleEnd(Http1xClientConnection.java:709)
        at io.vertx.core.impl.ContextImpl.execute(ContextImpl.java:327)
        at io.vertx.core.impl.DuplicatedContext.execute(DuplicatedContext.java:158)
        at io.vertx.core.http.impl.Http1xClientConnection.handleResponseEnd(Http1xClientConnection.java:940)
        at io.vertx.core.http.impl.Http1xClientConnection.handleHttpMessage(Http1xClientConnection.java:810)
        at io.vertx.core.http.impl.Http1xClientConnection.handleMessage(Http1xClientConnection.java:774)
        at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:159)
        at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:153)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:1583)


2024-12-16 12:30:36,288 INFO  [io.qua.lan.ope.com.OpenAiRestApi$OpenAiClientLogger] (vert.x-eventloop-thread-3) Response:
- status code: 200
- headers: [Content-Type: text/event-stream], [Date: Mon, 16 Dec 2024 17:30:36 GMT], [Transfer-Encoding: chunked]
- body: data: {"id":"chatcmpl-820","object":"chat.completion.chunk","created":1734370236,"model":"llama3.2","system_fingerprint":"fp_ollama","choices":[{"index":0,"delta":{"role":"assistant","content":"","tool_calls":[{"id":"call_hb8xgssf","index":0,"type":"function","function":{"name":"updateClaimStatus","arguments":"{\"claimId\":\"1\",\"status\":\"denied\"}"}}]},"finish_reason":null}]}

data: {"id":"chatcmpl-820","object":"chat.completion.chunk","created":1734370236,"model":"llama3.2","system_fingerprint":"fp_ollama","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":"stop"}]}

data: [DONE]

Please complete the following information:

  • LangChain4j version: 0.36.2
  • LLM(s) used: Ollama via OpenAI API
  • Java version: 21
  • Spring Boot version (if applicable):
@omarmahamid
Copy link
Contributor

omarmahamid commented Dec 17, 2024

@edeandrea

I did same steps to reproduce , couldn't reproduce it

what I'm missing

image

logs:

2024-12-17 22:07:59,116 INFO  [io.qua.lan.ope.com.OpenAiRestApi$OpenAiClientLogger] (vert.x-eventloop-thread-4) Request:
- method: POST
- url: http://localhost:11434/v1/embeddings
- headers: [Accept: application/json], [Authorization: Be...me], [Content-Type: application/json], [User-Agent: langchain4j-openai], [content-length: 1168]
- body: {
  "model" : "snowflake-arctic-embed",
  "input" : [ "    Claim ID: 1\n\n    Policy Inception Date: 1954-09-30\n\n    Claim Summary:\n    \n    On January 2nd, 1955, at around 3:30 PM, a car accident occurred at the intersection of Colima Road and Azusa Avenue in Hill Vallet. The involved parties were Marty McFly, driving a silver Delorean DMC-12 (OUTA-TIME), and Biff Tanner in a blue Type 2 Volkswagen Bus (BIF-RULZ).\n\n    Marty was heading south on Colima Road when Biff failed to stop at the red traffic signal on Asuza Avenue, causing a collision with Marty's vehicle. Both drivers exchanged information and took photos of the accident scene, which included damages to the front driver and passenger side of Marty's Delorean DMC-12 and the front driver's side of Biff's Volkswagen Bus. No injuries were reported.\n\n    Marty has attached necessary documents, such as photos, a police report, and an estimate for repair costs, to his email. He requests prompt attention to the claim and is available at (916) 555-4385 or marty.mcfly@email.com for any additional information or documentation needed.\n    \n\n    Question: Update claim status to denied\n" ]
}

what i'm missing ?

langchain4j added a commit that referenced this issue Dec 17, 2024
@langchain4j
Copy link
Owner

@edeandrea thank you for reporting!
@omarmahamid thank you for looking into it!

I was able to reproduce the issue (although in a different way, but I think it is the same issue) and fix it in d2fe663.

langchain4j added a commit that referenced this issue Dec 17, 2024
@langchain4j
Copy link
Owner

I did a mistake, the fix is also in fef358a

Repository owner deleted a comment from dosubot bot Dec 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Streaming Tools
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants