Connection between HA and Node Red keep Resetting #1335
Comments
|
What do the Home Assistant logs show anything at the time of the disconnects? |
|
Just now, I reproduced this issue, it seems that there is NO obvisous error repoting on home asssistant side when nodered reporting errors: here is the logs from nodered: and below are the logs from home assistant at the same time( according to timestamp), which is just a warning and has nothing to do with nodred: my gut feeling is: this issue come out when I use this: but that's just my gut feeling |
|
I have 20 switches configured and they are becoming unavailable every 2-4 minutes. |
|
Actually, before the crash, I cal also see a warning: Possible EventEmitter memory leak detected. 11 flows:started listeners added to [EventEmitter]. Use emitter.setMaxListeners() to increase limit |
|
Actually, before the connection between nodered and HA crashes, I can see that it takes several seconds to send a message from nodered to HA. I made a screen shot from node red: you can see that the "call service" node could be staying "sending" status for so long time that I have enough time to make a screenshot. Shortly after that, the connection crashes. This problem happens so often, and it's not difficult for to reproduce it. I can support the debug activities if necessary.
|
|
I am seeing a similar issue #1327 . At some point of time NR logs that the connection is closed. Due to that the sensor entity nodes that expect to send data to HA fail and throw an error. In HA no error is logged at the same time. |
|
I am seeing a noticable period after executing a Call Service when the node shows its status as "disconnected", before the connection is automatically reconnected. Nothing in most of the logs ... but I did find |
Hi @donburch888 , Are you running node red in a standalong docker (from homeassistant) container or you're using HASS OS? I'm wondering whether I should go back HASS OS setup ... This issue is really annoying, it's happening all of the time, and make node red almost useless for home assistant... |
|
I am now running HAOS under proxmox, so I could run things as separate VMs ... but no, I have just taken the easy way of running everything as HAOS add-ons. |
|
If you connect directly to Home Assistant and do not use the Supervisor proxy does that change anything? Uncheck use the add-on and create a long-lived access token. |
|
For anyone else wanting to follow zachowj's suggestion...
My test flow is working ... quickly pressing the inject node to say "this is a test" is no longer requiring a pause to let the server reconnect ... and back in HA, the Node-RED Log is no longer disconnecting and reconnecting. :-)
Looking good for me. |
I was using long-lived access token, and the checkbox of "using home assistant addon" was always unchecked. Hi, @zachowj, as this issue is quite easy to reproduce on my setup, do you think it makes sense that you create a debug version (docker), which could provided more logs hopefully, and I run it on my machine, and send you back those log file? and I could assist you closely to find the root cause of this issue. currently, I'm using nodered on my k8s cluster. |
|
@taoyx Do you use render template nodes? |
|
Hi @zachowj, Thank you. I guess you are talk about this node:
the attached block are the only one in my application: I just did a try, but didn't really use it. Now let me delete it, and see whether it makes a difference. Also, I have a gut feeling: the problems come offen when the resource(cpu, memory) pressure comes. But that's just a feeling, because, when I assign the home assistant container and node red on to a 2-core/4G memory kubernetes node, the problem is still happending. Also, I found nodered did eat quite a lot memory on my k8s cluster:
Maybe I'm too verbose... hope it could help a little bit. Thank you very much.
|
|
Hi @zachowj, Hi After I removed those template node from nodered, I didn't see anychange: the connection between HA and Nodered is still resetting. I keep seeing this warning of memory leakage, do you know how to debug it? what is (node:6) below? (node:6) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 flows:started listeners added to [EventEmitter]. Use emitter.setMaxListeners() to increase limit thank you,
|
|
when I go through the remaining issue, there are several duplicate one. From my point, this issue is the most painfule one for this project: It happens several times a day, and make the automation very unreliable. Shall we looking for alternative solution? |
|
I really suspect this issure is related with memory leak: that's reason there are no much use logs either on HA side or NR side. Normally, before the issue happens, we can always see such warning message: (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 flows:started listeners added to [EventEmitter]. Use emitter.setMaxListeners() to increase limit the similiar issue happens in another project, and they seems found the issue, refer here: quoto here: In fixResponseChunkedTransferBadEnding() two signals are added to the socket in every request, but they are never removed. In node-fetch/node-fetch#1474 I fix that problem by removing both signals on the socket's close() event. Hope it helps. @zachowj PS. also attach my latest logs: |
|
I'm suspecting a memory leak as well. I woke up a couple days ago to find the machine I run node-red on (alongside other services) completely locked up, and had to reboot it. I track memory usage for the machine running these services in home assistant, and it was at 100% prior to freezing. While node-red is not always pegging the system at 100% when the connection/disconnection issue occurs, I do see slowly rising memory usage until node-red is rebooted, at which point usage returns to normal. For me, the disconnection from HA seems to happen about once an hour. It's during this period that memory usage begins to rise. No logs on HA's side, though the Node-RED container logs just repeat as such until I restart the container: Seems related (if not the same) to this issue as well: #1225 I should note that I've tried downgrading the version of Node-RED & |
|
Hello, zachowj, Do you think this post could be relavent to this issue? https://community.home-assistant.io/t/node-red-losing-websocket-connection/159372/42 thanks, |
|
This issue has been automatically marked as abandoned because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
Describe the bug
from time to time, the connection between HA and Nodered just keep resetting: from within node red, the HA(ex. Event State node) status just going from "connnecting, connected, running... connecting, connnected, running... )
To Reproduce
It's quite difficult to reproduce it. it just happens quite randomly. the frequency is in the last several month, it happens quite offen. the frequency also varies: sometime, it happen very often, such as , for the whole night, I guess), it keep resetting; some time it LOOKS stable. But generally speaking, with my set up, it happens for every 2 or three days.
I have a feeling that the error was triggered by a wrong "call service" action. but when I purposely trigger a wrong "call service" action, it does not necessary trigger this error.
Please take a look at the logs I attached here.
I have a quite complicated node red flows: about 20 flows, and 4000 nodes.
nodered.log
Expected behavior
The connection between HA and Node red should maintained.
Screenshots
this issue simply happends on every flow which as a node from Homeassistant.
Example Flow
[ { "id": "ac7f510db8bb03d7", "type": "subflow", "name": "log", "info": "", "category": "", "in": [ { "x": 140, "y": 180, "wires": [ { "id": "ec6d8e0ab1345d42" } ] } ], "out": [ { "x": 760, "y": 180, "wires": [ { "id": "ec6d8e0ab1345d42", "port": 0 } ] } ], "env": [ { "name": "log_attribute", "type": "str", "value": "" } ], "meta": {}, "color": "#DDAA99", "status": { "x": 760, "y": 320, "wires": [ { "id": "a005a35b4ea9adf8", "port": 0 } ] } }, { "id": "a005a35b4ea9adf8", "type": "change", "z": "ac7f510db8bb03d7", "name": "", "rules": [ { "t": "set", "p": "payload", "pt": "msg", "to": "log_display", "tot": "msg" } ], "action": "", "property": "", "from": "", "to": "", "reg": false, "x": 600, "y": 300, "wires": [ [] ] }, { "id": "ec6d8e0ab1345d42", "type": "function", "z": "ac7f510db8bb03d7", "name": "", "func": "var attributeName = env.get(\"log_attribute\");\nvar timestamp = new Date().toLocaleString('en-GB');\n\nif (attributeName && attributeName.trim() !== \"\") {\n var attributeValue = msg[attributeName];\n msg.log_display = `${timestamp}: ${attributeValue}`;\n} else {\n msg.log_display = timestamp;\n}\n\nreturn msg;\n\n\n\n\n\n\n\n\n// var attributeName = env.get(\"log_attribute\");\n\n// var attributeValue ;\n\n// var timestamp = (new Date()).toLocaleDateString() + \" \" + (new Date()).toLocaleTimeString('en-GB');\n\n\n// if (attributeName !== undefined && attributeName.trim() !== \"\") {\n\n\n// attributeValue = msg[attributeName];\n// msg.log_display = timestamp + \": \" + attributeValue\n\n// } else {\n// msg.log_display = timestamp \n// }\n\n\n\n\n// return msg;\n\n", "outputs": 1, "noerr": 0, "initialize": "", "finalize": "", "libs": [], "x": 340, "y": 180, "wires": [ [ "a005a35b4ea9adf8" ] ] }, { "id": "34dc5a14e791abf6", "type": "link in", "z": "0266edf078264249", "name": "in: refreshing", "links": [ "5a02b52f1957efea", "66ed5db38f9e2acf", "690f2a19c27af2d8" ], "x": 415, "y": 1020, "wires": [ [ "88af364cf06e07f7" ] ] }, { "id": "2a58a7a5e14b3035", "type": "subflow:ac7f510db8bb03d7", "z": "0266edf078264249", "name": "sync statemachine with current_time slot", "x": 840, "y": 980, "wires": [ [ "25563b0f59368225" ] ] }, { "id": "b5719cd394eef0ed", "type": "switch", "z": "0266edf078264249", "name": "from_normal_trigger", "property": "from_normal_trigger", "propertyType": "flow", "rules": [ { "t": "true" }, { "t": "else" } ], "checkall": "false", "repair": false, "outputs": 2, "x": 720, "y": 840, "wires": [ [ "514eb5393137b402" ], [ "4add15f3bc3397dd" ] ] }, { "id": "514eb5393137b402", "type": "subflow:ac7f510db8bb03d7", "z": "0266edf078264249", "name": "", "x": 750, "y": 760, "wires": [ [] ] }, { "id": "76fae01a5df262f5", "type": "server-state-changed", "z": "0266edf078264249", "name": "current_time_slot -- no-delete", "server": "538475ac.a2270c", "version": 5, "outputs": 1, "exposeAsEntityConfig": "", "entityId": "input_select.current_time_slot", "entityIdType": "exact", "outputInitially": false, "stateType": "str", "ifState": "", "ifStateType": "str", "ifStateOperator": "is", "outputOnlyOnStateChange": true, "for": "", "forType": "num", "forUnits": "seconds", "ignorePrevStateNull": false, "ignorePrevStateUnknown": false, "ignorePrevStateUnavailable": false, "ignoreCurrentStateUnknown": false, "ignoreCurrentStateUnavailable": false, "outputProperties": [ { "property": "initial_time_slot", "propertyType": "msg", "value": "", "valueType": "entityState" } ], "x": 460, "y": 840, "wires": [ [ "b5719cd394eef0ed" ] ] }, { "id": "4add15f3bc3397dd", "type": "subflow:ac7f510db8bb03d7", "z": "0266edf078264249", "name": "", "x": 910, "y": 840, "wires": [ [ "25563b0f59368225" ] ] }, { "id": "88af364cf06e07f7", "type": "switch", "z": "0266edf078264249", "name": "running?", "property": "ha_status", "propertyType": "global", "rules": [ { "t": "eq", "v": "running", "vt": "str" }, { "t": "else" } ], "checkall": "true", "repair": false, "outputs": 2, "x": 520, "y": 1020, "wires": [ [ "2a58a7a5e14b3035" ], [ "877343a80b8c00e1" ] ] }, { "id": "671e455f6ebb0f5a", "type": "subflow:ac7f510db8bb03d7", "z": "0266edf078264249", "name": "", "env": [ { "name": "log_attribute", "value": "ha_status", "type": "str" } ], "x": 870, "y": 1060, "wires": [ [] ] }, { "id": "877343a80b8c00e1", "type": "change", "z": "0266edf078264249", "name": "ha_status", "rules": [ { "t": "set", "p": "ha_status", "pt": "msg", "to": "ha_status", "tot": "global" } ], "action": "", "property": "", "from": "", "to": "", "reg": false, "x": 740, "y": 1060, "wires": [ [ "671e455f6ebb0f5a" ] ] }, { "id": "25563b0f59368225", "type": "switch", "z": "0266edf078264249", "name": "running?", "property": "ha_status", "propertyType": "global", "rules": [ { "t": "eq", "v": "running", "vt": "str" }, { "t": "else" } ], "checkall": "true", "repair": false, "outputs": 2, "x": 1080, "y": 840, "wires": [ [ "ba6ac0826c84c1d0" ], [ "01fd36080f85fccb" ] ] }, { "id": "538475ac.a2270c", "type": "server", "name": "Home Assistant", "version": 5, "addon": false, "rejectUnauthorizedCerts": true, "ha_boolean": "y|yes|true|on|home|open", "connectionDelay": false, "cacheJson": true, "heartbeat": true, "heartbeatInterval": "30", "areaSelector": "friendlyName", "deviceSelector": "friendlyName", "entitySelector": "id", "statusSeparator": "at: ", "statusYear": "hidden", "statusMonth": "hidden", "statusDay": "2-digit", "statusHourCycle": "h23", "statusTimeFormat": "h:m", "enableGlobalContextStore": true } ]Environment Information
Version:
Home Assistant version: 2024.3.1 (I'm using Kubernetes/Docker)
Companion version: v3.1.3
Node-RED version: V3.1.7 (also in Docker i mage: nodered/node-red:3.1.7)
Docker: Yes (Kubernetes)
Add-on: N.A
Node.js version: N.A (Node.js in Docker?)
OS: Ubuntu 23.10
Additional context
No response
The text was updated successfully, but these errors were encountered: