DevOps Zone is brought to you in partnership with:

Mitch Pronschinske is the Lead Research Analyst at DZone. Researching and compiling content for DZone's research guides is his primary job. He likes to make his own ringtones, watches cartoons/anime, enjoys card and board games, and plays the accordion. Mitch is a DZone Zone Leader and has posted 2576 posts at DZone. You can read more from them at their website. View Full User Profile

DevOps Tool - Graph Processing Like a G2

11.04.2011
| 3699 views |
  • submit to reddit
What's should an administrator to do when the convoluted stream of log files from your new distributed system is just too overwhelming.  Abstraction is the answer according to a new research paper (via Microsoft) that suggests you use "execution graphs that model runtime events and their correlations in distributed systems." There is a graph processing tool for this called G2, and it specializes in diagnosing distributed systems.  Here's a bit of the abstract:

It works on execution graphs that model runtime events and their correlations in distributed systems. In G2, a diagnosis process involves a series of queries, expressed in a high-level declarative language that supports both relational and graph-based operators. Each query is compiled into a distributed execution. G2’s execution engine supports both parallel relational dataprocessing and iterative graph traversal.

Execution graphs in G2 tend to have long paths and are in structure distinctly different from other largescale graphs, such as social or web graphs. Tailoredfor execution graphs and graph traversal operations onthose graphs, G2’s graph engine distinguishes itself byembracing batched asynchronous iterations that allowsfor better parallelism without barriers, and by enabling partition-level states and aggregation.

We have applied G2 to diagnosis of distributed systems such as Berkeley DB, SCOPE/Dryad, and G2 itself to validate its effectiveness. When co-deployed on a 60-machine cluster, G2’s execution engine can handle execution graphs with millions of vertices and edges; for instance, using a query in G2, we traverse, filter, and summarize a 130 million-vertex graph into a 12 thousand vertex graph within 268 seconds on 60 machines. The use of an asynchronous model and a partition-level interface delivered a 66% reduction in response time when applied to queries in our diagnosis tasks.


Todd Hoff on High Scalability had some thoughts on this tool saying that the main downside would be issues arising from viewing applications as execution graphs when those programs have been written with low level code.  But there certainly is an upside: "There are many advantages to this approach, fault diagnosis and performance monitoring are just one of the wins."


Source: http://highscalability.com/blog/2011/11/3/paper-g2-a-graph-processing-system-for-diagnosing-distribute.html