|
|
|
Table of Contents Abstract We think the best way to get a sense of the power and flexibility behind Webob is to see it in action. The Webob Demonstration Portal is a showcase for just a few of the great things you can do with Webob. Everything on the portal page is live and done without any additional commerical software (we do leverage some free, open source tools and describe those along the way). In this document we'll reference pieces off the configuration file; from the portal page itself you can see all the full configuration, as well as the stylesheets used to transform the xml into the dashboard interface. Have questions or need a little help with your Webob scripts? Done something cool with Webob? Just drop us a line --- we live for this stuff. Important information lives everywhere. Grand unification schemes are well and good, but the reality is that the data we care about is scattered all over the place. Webob provides an ideal platform to "tame" that data and bring it together into a single XML document. Web sites, server vitals, news headlines, weather, stocks, anything that you can reach out to with a URL or a script can be intergrated into one place; for the demo portal that unified view lives here. Once you've got a structured view of your world, you can do all kinds of great stuff. Webob can monitor that view for key conditions and thresholds, from simple failure to detailed business rules. Webob can notify the right people when triggers hit, maintain a history, take arbitrary corrective action, you name it. You can also transform that XML into any output format you like. The most common is an HTML-based dashboard, but output can be tuned for mobile or character mode devices, richly-formatted reports for management, RSS for syndication purposes, whatever. XSL transformation is the key to that flexibility. Each section below will examine one of the demonstration "panes". We've tried to include examples of as many Webob features as possible; but 1024x768 pixels just isn't enough room! CPU load, memory, disk space --- there's a body of low-level server data that everybody needs to keep track of. Webob can do this in a number of ways. On Windows, performance counters are fully integrated as a resource "type", with their own thresholding syntax that you can read about in the online documentation. For Linux or other systems, you have a few alternatives. The most traditional approach is to use SNMP to collect the data; Webob can easily integrate SNMP information using the open source net-snmp tools and a script such as xmlsnmp.pl --- this is how the data for the machine "aurora" in the Vital Signs panel is implemented. Example 1. Using snmpget and xmlsnmp.pl to fetch structured SNMP variables <resource name="xmlsnmp-aurora" type="process" include-body="xml">
<path><![CDATA[ /home/sean/webob/scripts/xmlsnmp.pl aurora ]]></path>
<xpath-must-match info="less that 2mb free memory">
//var[contains(name, 'memTotalFree') and value > 2048]
</xpath-must-match>
</resource>Implementing native SNMP collection is high on our list of priorities, so you can expect it in a version of Webob soon. The aurora example also shows how you can use XPath content matching to test for arbitrary conditions --- if the variable memTotalFree drops below 2048 bytes, the resource will trigger an error. In this case we have not enabled email alerts, but the value will show up as red on the portal display. You can also parse the output of a shell tools like "top" to fetch machine vital signs. We've built a script called topxml.awk to illustrate this approach; you can see examples on the portal for the machines "liberty" and "poetry". Example 2. Using ssh and shell tools to fetch server vitals <resource name="topxml-poetry" type="process" include-body="xml">
<path><![CDATA[ ssh poetry "top -p 0 -b -n1" |
/home/sean/webob/scripts/topxml.awk ]]></path>
<xpath-must-match info="memory use exceeds 95%">
//memory[((available - free) * 100 div available) < 95]
</xpath-must-match>
</resource>These examples also demonstrate the use of ssh client keys to securely run scripts on remote machines. Normally with ssh we would either need to interactively supply a password or hardcode one into our scripts; neither of which is a great option in a monitoring environment. We've set up a trusted relationship between the "webob" account on the monitoring server and one we've created on the remote server so that the password is not required. You can read more about ssh client keys and how to deploy them over at hacks.oreilly.com. In the vitals pane, you can click on the "liberty" machine name to see a popup of output from the "top" command. The "include-body" attribute is used with a value of "cdata" to capture all output of the command into the resource body element. The output of any command or script can be captured this way and made available on a dashboard, without a shell connection or adding additional load to your servers. The "jsify" template does the job of taking the CDATA text and turning it into a legal javascript string. This allows us to create popups dynamically, which is nice but can be a bit tricky. Example 3. Turning element text into a legal Javascript string <xsl:template name="jsify">
<xsl:param name="sz" />
<xsl:variable name="szBefore" select="substring-before($sz, '
')" />
<xsl:variable name="szAfter" select="substring-after($sz, '
')" />
<xsl:value-of select="$szBefore" />
<xsl:if test="$szAfter != ''">
<br/>
<xsl:call-template name="jsify">
<xsl:with-param name="sz" select="$szAfter" />
</xsl:call-template>
</xsl:if>
</xsl:template>
...
<xsl:call-template name="jsify">
<xsl:with-param name="sz"
select="translate(../resource[name='top-liberty']/body, '"
', ' ')" />Monitoring web sites is Webob's bread and butter. The Web Monitoring panel exhibits a few of the different things you can do with web content. The idea here is to think creatively --- almost any information you want is available on the Internet or your intranet, and Webob gives you the tools to harness and manage it. The Software Poetry corporate website is pretty simple --- but it needs to be running or we're not doing business. The portal makes sure that the site is responding and keeps a history of all outages. If the site is ever down for more than an hour, we get a discount from our hosting provider. We use the "skip" attribute so that after an hour of sustained outage email is sent telling them to pay up! Example 4. Basic monitoring with history and alerts <action name="logfile" type="file">
<file>/home/sean/webob/web/sp-outages.csv</file>
</action>
<action name="email-sp" type="email">
<address>webob@softwarepoetry.com</address>
</action>
...
<resource name="web-sp" type="url">
<path>http://www.softwarepoetry.com/default.asp</path>
<passthrough tag="ps-title">Software Poetry website</passthrough>
<interval>120</interval>
<do-action name="logfile" skip="1"/>
<do-action name="email-sp" skip="30" />
</resource>The history for Software Poetry is kept in a CSV file; you can download it by clicking the [history] link next to Software Poetry on the portal. History can also be written to a database --- details are in the online documentation. Also interesting is the "passthrough" element. As the name implies, this tag is simply inserted directly into the status xml, where it can be referenced by the XSL stylesheet that creates the dashboard interface. We sell another product, KBnow, that helps create and maintain web-based knowledgebases. Part of our marketing strategy is to always be in the first page of Google results for the query "knowledgebase software". Maintaining positioning requires an ongoing effort to make sure that our pages are optimized for relevance and page rank. Webob can make sure that we don't fall off of the first page, and let us know by email if we do. Example 5. Ensure Google ranking for a search term <resource name="web-yahoo" type="url">
<path>http://search.yahoo.com/bin/search?p=knowledgebase+software</path>
<passthrough tag="ps-title">Google ranking for 'knowledgebase software'</passthrough>
<must-contain>www.softwarepoetry.com/kbnow/</must-contain>
<interval>43200</interval>
<do-action name="email-sp" />
</resource>Notice that our "interval" for this resource is quite long; 43,200 seconds equals twelve hours. Google doesn't like automated systems thrashing its servers, and it makes little sense to do this test frequently. Idealab! has been a favorite whipping boy of the dot-com-disaster site F***edCompany.com for a long time. It probably would make sense for the folks there to use Webob to see when they're being talked about. We've configured the portal to trigger an event every time that the FC homepage contains the word "idealab". Example 6. Watch for particular terms on a web site <resource name="web-fc" type="url">
<path>http://www.fuckedcompany.com</path>
<passthrough tag="ps-title">Idealab mention on fc.com</passthrough>
<must-not-contain>idealab</must-not-contain>
<interval>21600</interval>
</resource>We thought this was a pretty funny example. More serious applications might be watching headline news for mentions of relevant issues, or keeping track of competitors that may be talking about you. Our final web example shows how you can use XPath content matching to do more sophisticated analysis than just matching text. Our Microsoft resource will trigger events if the stock price ever drops below $25. Example 7. Tracking stock thresholds <resource name="web-msft" type="url">
<path>http://www.softwarepoetry.com/webob/doc/tutorial2.asp?symbol=msft</path>
<passthrough tag="ps-title">Microsoft stock threshold</passthrough>
<xpath-must-match info="msft < $25">
//stockquote[symbol = 'MSFT' and quote >= 25]
</xpath-must-match>
<interval>3600</interval>
</resource>Sometimes you really want to watch trends, and charts are a great way to do that. While Webob can't build graphs by itself, it's simple to integrate it with the fantastically useful rrdtool by Tobi Oetiker. rrdtool allows you to insert samples into a "round robin database" and then easily query the data out in graphical form. This is what we've done to measure "ping" response time from a number of sites around the world. The script for creating the database is create-rrd.sh; the script for updating data is update-rrd.py (written in Python). There are a number of Webob resources that implement the graphs; one for each site we're pinging and one to call the update script. Here's what they look like: Example 8. Charting ping response times with rrdtool <!-- Ping Australia (University of Sydney) -->
<resource name="aussie-ping" type="process" include-body="xml">
<passthrough tag="ping-caption">Australia</passthrough>
<path>/usr/local/bin/xmlping www.usyd.edu.au</path>
<interval>120</interval>
</resource>
<!-- RRD Update Script -->
<resource name="update-rrd" type="process">
<timeout>180</timeout>
<path>/home/sean/webob/scripts/update-rrd.py</path>
<interval>120</interval>
</resource>Monitoring with ping is something we talk about on our Tips and Scripts page. In this case we've chosen to use our custom C++ pinger (source code for this tool is available from the tips page), but you could use the system ping commmand just as easily. Using an "update" resource turns out to be a very useful Webob trick. While it offers much less capability in terms of scheduling than something like cron, Webob can be a great place to run periodic jobs and make sure they complete successfully. It's certainly much simpler to have all the scripts related to Webob in one place. This sample only scratches the surface of what you can put together with Webob and rrdtool. I'd encourage you to learn more, especially if you're building a Webob dashboard for management types --- never underestimate the power of pretty graphs on management! Even if you have no "monitoring" requirements whatsoever, Webob provides a great platform for integrating XML data into one place. We've been amazed to find how Webob is being leveraged in different organizations. Tell us what you're doing! RSS is an xml-based syndication format that has really caught fire recently. RSS simply allows a web site to publish its content in a structured way for consumption by a "reader" similar to those used for Usenet newsgroups. Virtually any data you could want is now available as RSS: headline news, security alerts, tech updates, Amazon book lists, you name it. Because RSS is just XML, it can easily be integrated into a Webob dashboard, as we've done with the headline feed from internetnews.com. Example 9. Integrating an RSS feed into Webob <resource name="internet-news" type="url" include-body="xml">
<passthrough tag="ps-hilite">Microsoft</passthrough>
<path>http://headlines.internet.com/internetnews/top-news/news.rss</path>
<interval>3600</interval>
</resource>The key here is the now-familiar "include-body" resource attribute. This takes the XML found at the resource's URL and injects it directly into the Webob status XML. You can then use a bit of XSL to format the content for your dashboard. Remember that the full stylesheets that make up the demonstration portal are available for you to use however you like. One cool thing we've done with our RSS feed display is to use the XSL "contains" function to determine if a news item references Microsoft. If it does, we hilight that headline by displaying it in red text. You can do the same for terms that you care about, or even completely filter out ones that you're not interested in. Example 10. Transforming RSS into HTML with hilighting <xsl:template match="resource" mode="news">
<xsl:variable name="hilite" select="ps-hilite" />
<ul>
<xsl:for-each select="body/RDF/item">
<xsl:element name="li">
<xsl:element name="a">
<xsl:if test="contains(title, $hilite) or contains(description, $hilite)">
<xsl:attribute name="class">hilite</xsl:attribute>
</xsl:if>
<xsl:attribute name="href"><xsl:value-of select="link"/></xsl:attribute>
<xsl:value-of select="title" />
</xsl:element>
<br/>
<xsl:value-of select="description" />
</xsl:element>
</xsl:for-each>
</ul>
</xsl:template>We're really excited about this feature of Webob. Much of the useful data on the web and in enterprises is accessible in "semi-structured" format. For example, the folks at Symantec provide a great service listing the top active virus/worm threats at http://securityresponse.symantec.com. The list at the bottom of the page provides a threat level, discovery date, name and url for these threats. It's a snap to take this semi-structured data and turn it into completely structured data using regular expression transformation. Writing regular expressions can be a bear, but they are tremendously powerful; we recommend a quick web search for "regular expression tutorial" if they're new to you. The <regex-transform> element tells Webob to apply the regular expression to the resource output and turn it into a <re-transformed> xml fragment. This fragment will have one <re-match> child for each match found in the output. Under each <re-match> element is a set of <re-submatch> elements, one for the text that matched and one for each parenthesized group found in the expression. That's quite a mouthful ... the below example shows how it works for the securityresponse threat list: Example 11. Transforming html with a regular expression <resource name="top-virus-threats" type="url" include-body="xml">
<path>http://securityresponse.symantec.com</path>
<regex-transform><![CDATA[category_([0-9]).gif" width=24 height=24 border=0></a></td>[^<]*<td><p>
<a href="([^"]+)"[^>]*>[ \t\n\r]*([^<]+)</a></p></td>[^<]*<td>([^<]*)</td>]]></regex-transform>
<interval>3600</interval>
</resource>This regular expression (note that the linebreak in the expression is for readability only!) results in status xml similar to the following: <body parse-status="0">
<re-transformed>
<re-match>
<re-submatch index="0">category_4.gif" width=24 height=24 border=0></a>
</td><td><p><a href="/avcenter/venc/data/w32.sobig.f@mm.html"
onmouseover="document.riskT1.src='/avcenter/graphics/ssrc/category_4_on.gif'"
onmouseout="document.riskT1.src='/avcenter/graphics/ssrc/category_4.gif'">
W32.Sobig.F@mm</a></p></td><td>August 18, 2003</td></re-submatch>
<re-submatch index="1">4</re-submatch>
<re-submatch index="2">/avcenter/venc/data/w32.sobig.f@mm.html</re-submatch>
<re-submatch index="3">W32.Sobig.F@mm</re-submatch>
<re-submatch index="4">August 18, 2003</re-submatch>
</re-match>
...Wow ... pretty ugly. But take a look at the <re-submatch> elements indexed 1-4. The data we care about has been broken out into structured xml elements that we can maniuplate however we like. For example, we could use xpath matching to alert whenever a category 4 threat becomes active. Regex transformation opens up a whole world of structured content without requiring cumbersome scripts. We love it! Unisys has provided a great service for retreiving local forecasts as XML. Integrating that into our dashboard with include-body is a snap. Because the service is occassionally unavailable, we've also hooked in a "backup" source that is slightly less complete, but will serve in a pinch. We use the XSL "choose" function to pick the right source based on availability. Example 12. Fetching the weather for Bellevue, WA (Linebreak in URL added for page layout only!) <resource name="weather" type="url" include-body="xml">
<path>http://weather.unisysfsp.com/PDCWebService/
WeatherServices.asmx/GetWeather?ZipCode=98005</path>
<interval>3600</interval>
</resource>The only tricky thing we do here is to turn some of the all-uppercase text into lowercase. The XSL "translate" function helps us out here, at least for ASCII text. Example 13. Transforming weather into lowercase HTML <xsl:template match="resource" mode="weather">
<xsl:variable name="lowercase" select="'abcdefghijklmnopqrstuvwxyz'" />
<xsl:variable name="uppercase" select="'ABCDEFGHIJKLMNOPQRSTUVWXYZ'" />
<table>
<xsl:for-each select="body/WeatherForecast/DayForecast/DailyForecast">
<tr>
<td><xsl:value-of
select="translate(Day, $uppercase, $lowercase)" /></td>
<td><xsl:value-of
select="translate(Forecast, $uppercase, $lowercase)" /></td>
</tr>
</xsl:for-each>
</table>
</xsl:template> |
|