Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

`Log streaming service has been terminated` happens a lot #1149

Open
hvt opened this issue Aug 5, 2020 · 3 comments
Open

`Log streaming service has been terminated` happens a lot #1149

hvt opened this issue Aug 5, 2020 · 3 comments

Comments

@hvt
Copy link

@hvt hvt commented Aug 5, 2020

What happened?

When using devspace dev, a lot of times the message Log streaming service has been terminated appears. Indeed, displaying of the logs in the terminal stops. Perhaps not a bug, but I was wondering what is happening here? And is there any way to fix it?

What did you expect to happen instead?

That logging of the project is not interrupted / terminated on the cli.

How can we reproduce the bug? (as minimally and precisely as possible)

This mainly happens when I have two devspace dev projects running simultaneously.

Local Environment:

  • DevSpace Version: 4.14.0
  • Operating System: linux
  • Deployment method: kubectl apply

Kubernetes Cluster:

  • Cloud Provider: other
  • Kubernetes Version:
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-26T03:47:41Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:51:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

/kind bug

@FabianKramm
Copy link
Member

@FabianKramm FabianKramm commented Aug 5, 2020

@hvt thanks for reporting this issue! The error message occurs if devspace somehow loses connection to the pods where the logs are streamed from. Losing the connection could have several reasons:

  • The pod is restarting for some reason (you can verify this via kubectl get pods)
  • Some kubernetes api servers have a timeout for idle connections and will kill the connection if this timeout is reached
  • There is an issue with the network which causes the connection loss

Having multiple devspace dev running shouldn't cause any issues with log streaming usually, are these projects running in the same namespace or have mutual deployments?

@hvt
Copy link
Author

@hvt hvt commented Aug 6, 2020

The projects are running in the same namespace.

Mmm, I will see what kubectl get pods gives, around the time this is happening. Will get back once it appears again :].

@hvt
Copy link
Author

@hvt hvt commented Sep 9, 2020

Hey @FabianKramm, this indeed happens when the pod is getting terminated and restarted because of failing liveness or readiness checks (your first point) because hey, we're developing and stuff breaks ;], or when I have a really long lasting devspace dev session that probably triggers a timeout at the Kubernetes API (your second point).

Is there anything that can be done about this? I.e. find the new deployed pod? Or is there missing metadata since the pod is not explicitly started by devspace itself, but by Kubernetes itself?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.