Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Not working in module mode #184

Open
kortschak opened this issue Oct 14, 2019 · 27 comments
Open

Not working in module mode #184

kortschak opened this issue Oct 14, 2019 · 27 comments
Assignees

Comments

@kortschak
Copy link
Contributor

@kortschak kortschak commented Oct 14, 2019

I am trying to run github.com/kortschak/graphprac on my local machine. I always run with module mode on, Go version 1.13.

When I try to run the first cell of the notebook at that repo I get a failure that the build with buildmode=plugin failed. When I execute this in a terminal to investigate I see that it's because the go tool expects the go.mod file to exist and bails otherwise. Creating a go mod file in that directory allows the build to complete and also allows the cell to execute on the notebook.

Perhaps the go.mod from imported packages' modules could be placed in the gomacro build paths.

At the very least, it should be noted that it might be necessary to invoke the server with GO111MODULE=off jupyter notebook.

@kortschak
Copy link
Contributor Author

@kortschak kortschak commented Oct 14, 2019

Note that gomacro as of cosmos72/gomacro@24fc920 executes the code that fails here without issue.

@mattn
Copy link
Contributor

@mattn mattn commented Oct 18, 2019

OT: gophernotes dropped support of Go 1.10. So we can remove vendor.

@cosmos72
Copy link
Member

@cosmos72 cosmos72 commented Nov 16, 2019

https://github.com/cosmos72/gomacro has a go.mod and is thus compiled with modules support,
but its runtime is not yet module aware - thus evaluating import will not find modules in $GOPATH/pkg/mod.

I am working to fix that.

@cosmos72 cosmos72 self-assigned this Nov 16, 2019
@cosmos72
Copy link
Member

@cosmos72 cosmos72 commented Nov 28, 2019

I added modules support to gomacro - i.e. import is now module aware.
ASAP I will update also gophernotes to use the latest gomacro.

@cosmos72
Copy link
Member

@cosmos72 cosmos72 commented Nov 28, 2019

Working on it, see #189
Also see #187 for a very similar issue

@cosmos72
Copy link
Member

@cosmos72 cosmos72 commented Dec 2, 2019

It should work now, waiting for @kortschak confirmation before closing

@mattn
Copy link
Contributor

@mattn mattn commented Dec 2, 2019

Question, when importing different version of module which is already used in gophernotes's dependencies, what you expect?

error loading plugin "/home/mattn/go/src/gomacro.imports/gonum.org/v1/plot/plot.so": plugin.Open("/home/mattn/go/src/gomacro.imports/gonum.org/v1/plot/plot.so"): plugin was built with a different version of package golang.org/x/image/math/fixed (previous failure)
@cosmos72
Copy link
Member

@cosmos72 cosmos72 commented Dec 2, 2019

I expect exactly the error you report, because plugin allows loading the same package multiple times only if they have exactly the same version.

I already thought about this problem in the past, and there are two complementary solutions:

  1. If gophernotes depends on a package foo, generate an import file for it in the 'imports' directory
  2. When compiling a plugin to load the symbols of a package bar, also cache the symbols of all its dependencies - a later import of one of the dependencies will be satisfied from the cache, without loading another plugin (which may have a different version, as you point out).

These are future improvements for gophernotes (1) and gomacro (2)

@kortschak
Copy link
Contributor Author

@kortschak kortschak commented Dec 2, 2019

This is fixed for me as noted in the PR.

@mattn
Copy link
Contributor

@mattn mattn commented Dec 3, 2019

If I want to use packages in the document and the packages are used in gophernotes, I should set GO111MODULE=off ?

@cosmos72
Copy link
Member

@cosmos72 cosmos72 commented Dec 3, 2019

Environment GO111MODULE is now ignored by gophernotes at runtime: internally, it always sets it to on before importing a package.
The underlying Gomacro has a user-controlled option to toggle GO111MODULE on/off but gophernotes has no mechanism to invoke it (yet).

Anyway there should be very few packages used in gophernotes without a corresponding import file in 'imports' directory:
at first glance, I'd say only github.com/pebbe/zmq4 and github.com/satori/go.uuid.
Their transitive dependencies are all in the standard library, and that's not an issue.

Maybe I am missing something?

@mattn
Copy link
Contributor

@mattn mattn commented Dec 4, 2019

I could solve issue that can't import packages which is used in gophernotes.

One another issue is that can't import packages in different versions. I'm trying to use my package https://github.com/mattn/go-pairplot and this import golang.org/x/image but gonum.org/v1/plot/vg also import golnang.org/x/image/font/fixed.

image

@mattn
Copy link
Contributor

@mattn mattn commented Dec 4, 2019

Hmm, importing "github.com/mattn/go-runewidth" fail.

@sbinet
Copy link
Member

@sbinet sbinet commented Dec 4, 2019

Not sure if that's related but 'gopls' also fails to import 'runewidth' (this happens when trying to 'gorename' something in one of my packages that happens to import a pkg that, down the line, imports that 'runewidth' package.)

@mattn
Copy link
Contributor

@mattn mattn commented Dec 4, 2019

Hmm, I played using go-runewidth for a little while but I don't see the fail.

@mattn
Copy link
Contributor

@mattn mattn commented Dec 5, 2019

Anyone could you please try import "github.com/mattn/go-plotlib" ?

@cosmos72
Copy link
Member

@cosmos72 cosmos72 commented Dec 5, 2019

import "github.com/mattn/go-plotlib" fails with

error loading plugin "/home/max/go/src/gomacro.imports/github.com/mattn/go-plotlib/go-plotlib.so": plugin.Open("/home/max/go/src/gomacro.imports/github.com/mattn/go-plotlib/go-plotlib"): plugin was built with a different version of package github.com/mattn/go-runewidth

The reason is the following:
the latest tagged version of github.com/mattn/go-plotlib is 0.0.6 i.e. commit faf47dabfcfc44f71e7bf404f8effc6ac29a6cc9, which depends on github.com/mattn/go-runewidth 0.0.7.

But gophernotes depends on the most recent tagged version of github.com/peterh/liner i.e. 1.1.0 which depends on github.com/mattn/go-runewidth 0.0.3.

So github.com/mattn/go-runewidth 0.0.3 is statically linked inside gophernotes, thus plugin refuses to load github.com/mattn/go-runewidth 0.0.7.

The only real solution is for @peterh to release a newer version of his github.com/peterh/liner that will depend on github.com/mattn/go-runewidth 0.0.7

@peterh
Copy link

@peterh peterh commented Dec 7, 2019

I'm disinclined to release a new version of liner every time a 3rd party package sneezes. Fortunately, it appears that this is not "the only real solution" for @cosmos72 .

If I understand the Go module system correctly, "0.0.3" really means "0.x.y, where x >= 0 and y >= 3". So it should allow you to specify require github.com/mattn/go-runewidth 0.0.7 //indirect in your go.mod file (even if you don't actually use runewidth yourself) in order to get the specific version you require. See https://blog.golang.org/using-go-modules#TOC_4.

@cosmos72
Copy link
Member

@cosmos72 cosmos72 commented Dec 7, 2019

Thanks for the suggestion, I will try it.
I hope to be mistaken, because I remember about versions v0.x.y being treated as incompatible - maybe such rule exists in some other system?

@cosmos72
Copy link
Member

@cosmos72 cosmos72 commented Dec 8, 2019

I updated gomacro/go.mod to explicitly require github.com/mattn/go-runewidth 0.0.7 and it works.
I also updated gophernotes/go.mod to require the latest gomacro containing the change above.
Thanks @peterh

@cosmos72
Copy link
Member

@cosmos72 cosmos72 commented Dec 8, 2019

About the other issues listed above:

  • import "github.com/kortschak/graphprac" now works
  • import "gonum.org/v1/plot" now works too
  • import "github.com/mattn/go-pairplot" now works except if you already imported gonum.org/v1/plot in the same session, because they depend on different versions of golang.org/x/image/math/fixed. I don't know whether this is fixable by modifying only gophernotes - suggestions?
@cosmos72 cosmos72 closed this Feb 25, 2020
@RafalSkolasinski
Copy link

@RafalSkolasinski RafalSkolasinski commented May 4, 2020

I got similar issue when trying

import "github.com/seldonio/seldon-core/operator/apis/machinelearning.seldon.io/v1"
import "github.com/ghodss/yaml"

I get

error loading plugin "/home/rskolasinski/go/src/gomacro.imports/github.com/ghodss/yaml/yaml.so": plugin.Open("/home/rskolasinski/go/src/gomacro.imports/github.com/ghodss/yaml/yaml"): plugin was built with a different version of package gopkg.in/yaml.v2

It seems it tries to load from $GOPATH/src so not inmodules mode.
Is there a way to force usage of $GOPATH/pkg/mod when notebook is in directory containing go.mod file?

Sorry for bumping old issue - can open new one if this is some new problem.

@cosmos72
Copy link
Member

@cosmos72 cosmos72 commented May 6, 2020

Gophernotes should ignore the contents of current directory when importing packages:
the import machinery is executed in a newly created directory `$GOPATH/src/gomacro.imports/PACKAGE/FULL/PATH'

Maybe both packages depend on gopkg.in/yaml.v2 - and on different versions of it?
The import machinery is not yet smart enough to tell Go compiler
"I already loaded version V1 of package X. When getting/compiling package Y which depends on X too, please try to use version V1 of package X instead of the default version, and also instead of the versioned dependency specified by package Y"

@cosmos72 cosmos72 reopened this May 6, 2020
@RafalSkolasinski
Copy link

@RafalSkolasinski RafalSkolasinski commented May 7, 2020

Gophernotes should ignore the contents of current directory when importing packages:
the import machinery is executed in a newly created directory `$GOPATH/src/gomacro.imports/PACKAGE/FULL/PATH'

Coming to Go from Python background, I used to do a lot of my development by writing Python code in .py files (even packages) and importing it directly in the notebook for testing how it works and fooling around.

In some regards you could say that Jupyter Notebook was my IDE. I hoped that Gophernotes could serve the same purpose for Go. Wonder if worth opening another issue for it?

@cosmos72
Copy link
Member

@cosmos72 cosmos72 commented May 7, 2020

There should be an open issue that, among other things, proposes to implement import "." to load the Go package in the current directory. Unluckily I cannot find it anymore.

Feel free to open a new issue on this topic. If you can explain in detail the behavior you expect, even better :)

@mattn
Copy link
Contributor

@mattn mattn commented Sep 4, 2020

Sorry off topic. Can I import specific version of package in notebook?

@cosmos72
Copy link
Member

@cosmos72 cosmos72 commented Sep 4, 2020

Sorry off topic. Can I import specific version of package in notebook?

Better to discuss this in a separate issue - I opened #218

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.