Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make remote FeatureGenerators #7

Open
matt-gardner opened this issue Jan 5, 2016 · 2 comments
Open

Make remote FeatureGenerators #7

matt-gardner opened this issue Jan 5, 2016 · 2 comments

Comments

@matt-gardner
Copy link
Owner

With large graphs (such as Freebase), it can take upwards of 10 minutes just to load the graph from disk and create the graph object. Maybe there are some things I can do in code to make that a bit quicker, but it still would be nice to only have it done once and be able to reuse a running graph server.

@matt-gardner
Copy link
Owner Author

As of 1/21/16, this actually works, but it's incredibly slow. I did a bunch of work trying to optimize graph loading, and the way the graph is stored in memory. So that should help a bit. There is one more optimization I want to try for loading the graph (store a binary file, and load that, instead of the GraphChi ascii version I currently use), but that doesn't help the server any. To make the server idea really feasible, I need to offload more of the computation to the graph, having the graph object compute paths and such. That would be a pretty big refactoring of the code, so I'm not going to do it any time soon.

@matt-gardner matt-gardner changed the title Get the graph server working. Push path computation to the graph, so a graph server would actually be feasible Jan 21, 2016
@matt-gardner matt-gardner changed the title Push path computation to the graph, so a graph server would actually be feasible Make remote FeatureGenerators May 18, 2016
@matt-gardner
Copy link
Owner Author

Actually, to implement what I want to do, I just need to make a FeatureGeneratorServer, instead of a GraphServer. You issue a query about a node pair to the server, it does the graph computation, and responds with a list of features. This is definitely feasible. I just don't have a particular need for it at the moment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant