<commit_before>r""":mod:`sider.ext.wsgi_referer_stat` --- Collecting referers using sorted sets
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This example will show you when and how to use sorted sets basically.
We will build a small WSGI middleware that simply collects all
:mailheader:`Referer` of the given WSGI web application.

WSGI and middlewares
--------------------

WSGI is a standard interface between web servers and Python web
applications or frameworks to promote web application portability
across a variety of web servers.  (If you are from Java think servlet.
If you are from Ruby think Rack.)

WSGI application is a callable object (function object).  It takes
two arguments: the first one is an ``environ`` dictionary which
contains request data e.g. ``PATH_INFO``, ``QUERY_STRING``,
``REQUEST_METHOD``.  The last one is a ``start_response`` function
to respond.  WSGI application returns an iterable object that
contains content body of response.

WSGI applications can be deployed into WSGI containers (server
implementations).  There are a lot of production-ready WSGI
containers.  Some of these are super fast, and some of others
are very reliable.  Check `Green Unicorn`_, uWSGI_, mod_wsgi_,
and so forth.

WSGI middleware is somewhat like decorator pattern for WSGI
applications.  Usually they are implemented by using nested
higher-order functions or classes with :meth:`~object.__call__()`
special method.

.. seealso::

   To learn more details about WSGI, read :pep:`333` and other related
   resources.  This tutorial doesn't deal with WSGI.

   :pep:`333` --- Python Web Server Gateway Interface v1.0
      This document specifies a proposed standard interface between
      web servers and Python web applications or frameworks, to promote
      web application portability across a variety of web servers.

   `Getting Started with WSGI`__ by Armin Ronacher
      Armin Ronacher, the author of Flask_, Werkzeug_ and Jinja_,
      wrote this WSGI tutorial.

   `A Do-It-Yourself Framework`__ by Ian Bicking
      Ian Bicking, the author of Paste_, WebOb_, lxml.html_ and
      FormEncode_, explains about WSGI apps and middlewares.

.. _Green Unicorn: http://gunicorn.org/
.. _uWSGI: http://projects.unbit.it/uwsgi/
.. _mod_wsgi: http://code.google.com/p/modwsgi/
__ http://lucumr.pocoo.org/2007/5/21/getting-started-with-wsgi/
.. _Flask: http://flask.pocoo.org/
.. _Werkzeug: http://werkzeug.pocoo.org/
.. _Jinja: http://jinja.pocoo.org/
__ http://pythonpaste.org/do-it-yourself-framework.html
.. _Paste: http://pythonpaste.org/
.. _WebOb: http://www.webob.org/
.. _lxml.html: http://lxml.de/lxmlhtml.html
.. _FormEncode: http://www.formencode.org/

Simple idea
-----------

The simple idea we'll implement here is to collect all :mailheader:`Referer`
and store it into a persistent storage.  We will use Redis as its persistent
store.  We want to simply increment the number of count if it duplicates.

Stored data will be like:

==================================  =====
Referer                             Count
==================================  =====
http://dahlia.kr/                   1
https://bitbucket.org/dahlia/sider  3
https://twitter.com/hongminhee      6
==================================  =====

We could use a hash here, but sorted set seems more suitable.
Sorted sets are a data structure provided by Redis that is basically
a set but able to represent duplications as its scores (:redis:`ZINCRBY`).

We can list a sorted set in asceding (:redis:`ZRANGE`) or
descending order (:redis:`ZREVRANGE`) as well.

.. seealso::

    `Redis Data Types`__
       The Redis documentation that explains about its data
       types: strings, lists, sets, sorted sets and hashes.

__ http://redis.io/topics/data-types

Prototyping with using in-memory dictionary
-------------------------------------------

First of all, we can implement a proof-of-concept prototype without Redis.
Although Python has no sorted sets, but here we could use :class:`dict`
instead. ::

    def referer_stat_middleware(app):
        assert callable(app)
        referer_set = {}
        def enchanted_app(environ, start_response):
            try:
                referer = environ['HTTP_REFERER']
            except KeyError:
                pass
            else:
                try:
                    referer_set[referer] += 1
                except KeyError:
                    referer_set[referer] = 1
            return app(environ, start_response)
        return enchanged_app

Middlewares usually are implemented as a class instead of nested higher-order
functions when it's not plain so much.  We can rewrite it by using a class
with :meth:`~object.__call__()` special method::

    class RefererStatMiddleware(object):
        '''A simple WSGI middleware that collects :mailheader:`Referer`
        headers.

        '''

        def __init__(self, application):
            assert callable(application)
            self.application = application
            self.referer_set = {}

        def __call__(self, environ, start_response):
            try:
                referer = environ['HTTP_REFERER']
            except KeyError:
                pass
            else:
                try:
                    self.referer_set[referer] += 1
                except KeyError:
                    self.referer_set[referer] = 1
            return self.application(environ, start_response)

It has some problems yet.  What are that problems?

1. WSGI applications can be deployed into multiple server nodes,
   or forked to multiple processes as well.  That means:
   :attr:`RefererStatMiddleware.referer_set` attribute can be
   split and not shared.

2. Increments of duplication counts aren't atomic.

3. Data will be lost when server process is terminated.

We can solve those problems by using Redis sorted sets instead of
Python in-memorty :class:`dict`.

Sider and persistent objects
----------------------------

It's a simple job, so we can deal with Redis commands by our hands.
However it's a tutrial example of Sider.  :-)  We will use Sider's
sorted set abstraction here instead.  It's more abstracted away and
easier to use!

Before touch our middleware code, the following session in Python
interactive shell can make you understand basic of how to use
Sider:

>>> from redis.client import StrictRedis
>>> from sider.session import Session
>>> from sider.types import SortedSet
>>> session = Session(StrictRedis())
>>> my_sorted_set = session.get('my_sorted_set', SortedSet)
>>> my_sorted_set
<sider.sortedset.SortedSet ('my_sorted_set') {}>

.. note::

   Did you face :exc:`~exceptions.ImportError`?

   >>> from redis.client import StrictClient
   Traceback (most recent call last):
     File "<console>", line 1, in <module>
   ImportError: No module named redis

   You probably didn't install Python redis_ client library.
   You can install it through :program:`pip`:

   .. sourcecode:: console

      $ pip install redis

   Or :program:`easy_install`:

   .. sourcecode:: console

      $ easy_install redis

Okay, here's an empty set: ``my_sorted_set``.  Let's add something to it.

>>> my_sorted_set
<sider.sortedset.SortedSet ('my_sorted_set') {}>
>>> my_sorted_set.add('kimchi')  # ZINCRBY
>>> my_sorted_set
<sider.sortedset.SortedSet ('my_sorted_set') {'kimchi'}>

Unlike Python's in-memory :class:`set` or :class:`dict`,
it's a persistent object.  In other words, ``my_sorted_set``
still contains ``'kimchi'`` even if you quit this session of
Python interactive shell.  Try yourself: type :data:`exit() <exit>`
to quit the session and enter :program:`python` again.  And then...

>>> my_sorted_set
Traceback (most recent call last):
  File "<console>", line 1, in <module>
NameError: global name 'my_sorted_set' is not defined

I didn't lie!  You have to load Sider session again first.

>>> from redis.client import StrictRedis
>>> from sider.session import Session
>>> from sider.types import SortedSet
>>> client = StrictRedis()
>>> session = Session(client)
>>> my_sorted_set = session.get('my_sorted_set', SortedSet)

Good, and then:

>>> my_sorted_set
<sider.sortedset.SortedSet ('my_sorted_set') {'kimchi'}>

Yeah!

Note that the following line:

>>> client = StrictRedis()

tries to connect to Redis server on localhost:6379 by default.
There are ``host`` and ``port`` parameters to configure it.

>>> client = StrictRedis(host='localhost', port=6379)

.. _redis: https://github.com/andymccurdy/redis-py

Sorted sets
-----------

You can :meth:`~sider.sortedset.SortedSet.update()` multiple values at a time:

>>> my_sorted_set.update(['bibimbap', 'bulgogi'])  # ZINCRBY
>>> my_sorted_set
<sider.sortedset.SortedSet ('my_sorted_set') {'bibimbap', 'bulgogi', 'kimchi'}>
>>> my_sorted_set.update(['kimchi', 'bibimbap'])  # ZINCRBY
>>> my_sorted_set
<sider.sortedset.SortedSet ('my_sorted_set')
 {'bulgogi', 'bibimbap': 2.0, 'kimchi': 2.0}>
>>> my_sorted_set['bibimbap']  # ZSCORE
2.0
>>> my_sorted_set.add('bibimbap')
>>> my_sorted_set['bibimbap']  # ZSCORE
3.0

As you can see, doubly added members get double scores.  This property is
what we will use in the middleware.

You can list values and these scores the sorted set contains.
Similar to :class:`dict` there's :meth:`~sider.sortedset.SortedSet.items()`
method.

>>> my_sorted_set.items()  # ZRANGE
[('bulgogi', 1.0), ('bibimbap', 2.0), ('kimchi', 2.0)]
>>> my_sorted_set.items(reverse=True)  # ZREVRANGE
[('kimchi', 2.0), ('bibimbap', 2.0), ('bulgogi', 1.0)]

There are other many features to :class:`~sider.sortedset.SortedSet` type,
but it's all we need to know to implement the middleware.  So we stop
introduction of the type to step forward.

Replace :class:`dict` with :class:`~sider.sortedset.SortedSet`
--------------------------------------------------------------

To replace :class:`dict` with :class:`~sider.sortedset.SortedSet`,
look :meth:`RefererStatMiddleware.__init__()` method first::

    def __init__(self, application):
        self.application = application
        self.referer_set = {}

.. note::

   The following codes implictly assumes that it imports::

       from redis.client import StrictClient
       from sider.session import Session
       from sider.types import SortedSet

The above code can be easily changed to:

::

    def __init__(self, application):
        assert callable(application)
        self.application = application
        client = StrictRedis()
        session = Session(client)
        self.referer_set = session.get('wsgi_referer_set', SortedSet)

It should be more configurable by users.  Redis key is currently hard-coded
as ``wsgi_referer_set``.  It can be parameterized, right? ::

    def __init__(self, set_key, application):
        assert callable(application)
        self.application = application
        client = StrictRedis()
        session = Session(client)
        self.referer_set = session.get(str(set_key), SortedSet)

It still lacks configurability.  Users can't set address of Redis server
to connect.  Parameterize ``session`` as well::

    def __init__(self, session, set_key, application):
        assert isinstance(session, Session)
        assert callable(application)
        self.application = application
        self.referer_set = session.get(str(set_key), SortedSet)

Okay, it's enough flexible to environments.  Our first and third problems
have just solved.  Its data become shared and don't be split anymore.
No data loss even if process has terminated.

Next, we have to make increment atomic.  See a part of
:meth:`RefererStatMiddleware.__call__()` method::

    try:
        self.referer_set[referer] += 1
    except KeyError:
        self.referer_set[referer] = 1

Redis sorted set offers a simple atomic way to increase its score:
:redis:`ZINCRBY`.  Sider maps :redis:`ZINCRBY` command to
:meth:`SortedSet.add() <sider.sortedset.SortedSet.add>` method.
So, those lines can be replaced by the following line::

    self.referer_set.add(referer)

and it will be committed atomically.

Source code
-----------

The complete source code of this example can be found in
:file:`examples/wsgi-referer-stat/` directory of the repository.

It's public domain, feel free!

Final API
---------

"""
from sider.session import Session
from sider.types import SortedSet
from jinja2 import Environment, PackageLoader

class RefererStatMiddleware(object):
    """A simple WSGI middleware that collects :mailheader:`Referer`
    headers and stores it into a Redis sorted set.

    You can see the list of referrers ordered by duplication count
    in ``/__stat__`` page (or you can configure the ``stat_path``
    argument).

    :param session: sider session object
    :type session: :class:`sider.session.Session`
    :param set_key: the key name of Redis sorted set
                    to store data
    :type set_key: :class:`basestring`
    :param application: wsgi app to wrap
    :type application: :class:`collections.Callable`
    :param stat_path: path to see the collected data.
                      default is ``'/__stat__'``.
                      if it's ``None`` the data cannot be accessed
                      from outside
    :type stat_path: :class:`basestring`

    """

    #: (:class:`sider.sortedset.SortedSet`) The set of collected
    #: :mailheader:`Referer` strings.
    referer_set = None

    def __init__(self, session, set_key, application, stat_path='/__stat__'):
        if not isinstance(session, Session):
            raise TypeError('session must be an instance of sider.session.'
                            'Session, not ' + repr(session))
        elif not callable(application):
            raise TypeError('application must be callable, not ' +
                            repr(application))
        self.session = session
        self.application = application
        self.referer_set = session.get(str(set_key), SortedSet)
        self.stat_path = stat_path
        loader = PackageLoader(__name__)
        environment = Environment(loader=loader)
        self.template = environment.get_template('stat.html')

    def stat_application(self, environ, start_response):
        """WSGI application that lists its collected referers."""
        content_type = 'text/html; charset=utf-8'
        start_response('200 OK', [('Content-Type', content_type)])
        referers = self.referer_set.items(reverse=True)
        stream = self.template.generate(referers=referers)
        for chunk in stream:
            yield chunk.encode('utf-8', 'ignore')

    def __call__(self, environ, start_response):
        if self.stat_path is not None:
            path = environ['PATH_INFO']
            if path == self.stat_path or path.startswith(self.stat_path + '/'):
                return self.stat_application(environ, start_response)
        try:
            referer = environ['HTTP_REFERER']
        except KeyError:
            pass
        else:
            self.referer_set.add(referer)
        return self.application(environ, start_response)

<commit_msg>Revised writings<commit_after>r""":mod:`sider.ext.wsgi_referer_stat` --- Collecting referers using sorted sets
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This tutorial will show you basic example of how to use sorted sets.
We will build a small WSGI middleware that simply collects all
:mailheader:`Referer` of the given WSGI web application.

WSGI and middlewares
--------------------

WSGI is a standard interface between web servers and Python web
applications or frameworks to promote web application portability
across a variety of web servers.  (If you are from Java think servlet.
If you are from Ruby think Rack.)

WSGI applications can be deployed into WSGI containers (server
implementations).  There are a lot of production-ready WSGI
containers.  Some of these are super fast, and some of others
are very reliable.  Check `Green Unicorn`_, uWSGI_, mod_wsgi_,
and so forth.

WSGI middleware is somewhat like decorator pattern for WSGI
applications.  Usually they are implemented by using nested
higher-order functions or classes with :meth:`~object.__call__()`
special method.

.. seealso::

   To learn more details about WSGI, read :pep:`333` and other related
   resources.  This tutorial doesn't deal with WSGI.

   :pep:`333` --- Python Web Server Gateway Interface v1.0
      This document specifies a proposed standard interface between
      web servers and Python web applications or frameworks, to promote
      web application portability across a variety of web servers.

   `Getting Started with WSGI`__ by Armin Ronacher
      Armin Ronacher, the author of Flask_, Werkzeug_ and Jinja_,
      wrote this WSGI tutorial.

   `A Do-It-Yourself Framework`__ by Ian Bicking
      Ian Bicking, the author of Paste_, WebOb_, lxml.html_ and
      FormEncode_, explains about WSGI apps and middlewares.

.. _Green Unicorn: http://gunicorn.org/
.. _uWSGI: http://projects.unbit.it/uwsgi/
.. _mod_wsgi: http://code.google.com/p/modwsgi/
__ http://lucumr.pocoo.org/2007/5/21/getting-started-with-wsgi/
.. _Flask: http://flask.pocoo.org/
.. _Werkzeug: http://werkzeug.pocoo.org/
.. _Jinja: http://jinja.pocoo.org/
__ http://pythonpaste.org/do-it-yourself-framework.html
.. _Paste: http://pythonpaste.org/
.. _WebOb: http://www.webob.org/
.. _lxml.html: http://lxml.de/lxmlhtml.html
.. _FormEncode: http://www.formencode.org/

Simple idea
-----------

The simple idea we'll implement here is to collect all :mailheader:`Referer`
and store it into a persistent storage.  We will use Redis as its persistent
store.  We want to simply increment the number of count if it duplicates.

Stored data will be like:

==================================  =====
Referer                             Count
==================================  =====
http://dahlia.kr/                   1
https://bitbucket.org/dahlia/sider  3
https://twitter.com/hongminhee      6
==================================  =====

We could use a hash here, but sorted set seems more suitable.
Sorted sets are a data structure provided by Redis that is basically
a set but able to represent duplications as its scores (:redis:`ZINCRBY`).

We can list a sorted set in asceding (:redis:`ZRANGE`) or
descending order (:redis:`ZREVRANGE`) as well.

.. seealso::

    `Redis Data Types`__
       The Redis documentation that explains about its data
       types: strings, lists, sets, sorted sets and hashes.

__ http://redis.io/topics/data-types

Prototyping with using in-memory dictionary
-------------------------------------------

First of all, we can implement a proof-of-concept prototype without Redis.
Although Python has no sorted sets, but here we could use :class:`dict`
instead. ::

    class RefererStatMiddleware(object):
        '''A simple WSGI middleware that collects :mailheader:`Referer`
        headers.

        '''

        def __init__(self, application):
            assert callable(application)
            self.application = application
            self.referer_set = {}

        def __call__(self, environ, start_response):
            try:
                referer = environ['HTTP_REFERER']
            except KeyError:
                pass
            else:
                try:
                    self.referer_set[referer] += 1
                except KeyError:
                    self.referer_set[referer] = 1
            return self.application(environ, start_response)

It has some problems yet.  What are that problems?

1. WSGI applications can be deployed into multiple server nodes,
   or forked to multiple processes as well.  That means:
   :attr:`RefererStatMiddleware.referer_set` attribute can be
   split and not shared.

2. Increments of duplication counts aren't atomic.

3. Data will be lost when server process is terminated.

We can solve those problems by using Redis sorted sets instead of
Python in-memorty :class:`dict`.

Sider and persistent objects
----------------------------

It's a simple job, so we can deal with Redis commands by our hands.
However it's a tutrial example of Sider.  :-)  We will use Sider's
sorted set abstraction here instead.  It's more abstracted away and
easier to use!

Before touch our middleware code, the following session in Python
interactive shell can make you understand basic of how to use
Sider:

>>> from redis.client import StrictRedis
>>> from sider.session import Session
>>> from sider.types import SortedSet
>>> session = Session(StrictRedis())
>>> my_sorted_set = session.get('my_sorted_set', SortedSet)
>>> my_sorted_set
<sider.sortedset.SortedSet ('my_sorted_set') {}>

.. note::

   Did you face :exc:`~exceptions.ImportError`?

   >>> from redis.client import StrictClient
   Traceback (most recent call last):
     File "<console>", line 1, in <module>
   ImportError: No module named redis

   You probably didn't install Python redis_ client library.
   You can install it through :program:`pip`:

   .. sourcecode:: console

      $ pip install redis

   Or :program:`easy_install`:

   .. sourcecode:: console

      $ easy_install redis

Okay, here's an empty set: ``my_sorted_set``.  Let's add something to it.

>>> my_sorted_set
<sider.sortedset.SortedSet ('my_sorted_set') {}>
>>> my_sorted_set.add('http://dahlia.kr/')  # ZINCRBY
>>> my_sorted_set
<sider.sortedset.SortedSet ('my_sorted_set') {'http://dahlia.kr/'}>

Unlike Python's in-memory :class:`set` or :class:`dict`,
it's a persistent object.  In other words, ``my_sorted_set``
still contains ``'kimchi'`` even if you quit this session of
Python interactive shell.  Try yourself: type :data:`exit() <exit>`
to quit the session and enter :program:`python` again.  And then...

>>> my_sorted_set
Traceback (most recent call last):
  File "<console>", line 1, in <module>
NameError: global name 'my_sorted_set' is not defined

I didn't lie!  You have to load Sider session again first.

>>> from redis.client import StrictRedis
>>> from sider.session import Session
>>> from sider.types import SortedSet
>>> client = StrictRedis()
>>> session = Session(client)
>>> my_sorted_set = session.get('my_sorted_set', SortedSet)

Good, and then:

>>> my_sorted_set
<sider.sortedset.SortedSet ('my_sorted_set') {'http://dahlia.kr/'}>

Yeah!

Note that the following line:

>>> client = StrictRedis()

tries to connect to Redis server on localhost:6379 by default.
There are ``host`` and ``port`` parameters to configure it.

>>> client = StrictRedis(host='localhost', port=6379)

.. _redis: https://github.com/andymccurdy/redis-py

Sorted sets
-----------

You can :meth:`~sider.sortedset.SortedSet.update()` multiple values at a time:

>>> my_sorted_set.update(['https://bitbucket.org/dahlia/sider',
...                       'https://twitter.com/hongminhee'])  # ZINCRBY
>>> my_sorted_set
<sider.sortedset.SortedSet ('my_sorted_set')
 {'https://bitbucket.org/dahlia/sider', 'https://twitter.com/hongminhee',
  'http://dahlia.kr/'}>
>>> my_sorted_set.update(['http://dahlia.kr/',
...                       'https://twitter.com/hongminhee'])  # ZINCRBY
>>> my_sorted_set
<sider.sortedset.SortedSet ('my_sorted_set')
 {'https://bitbucket.org/dahlia/sider', 'https://twitter.com/hongminhee': 2.0,
  'http://dahlia.kr/': 2.0}>
>>> my_sorted_set['http://dahlia.kr/']  # ZSCORE
2.0
>>> my_sorted_set.add('http://dahlia.kr/')
>>> my_sorted_set['http://dahlia.kr/']  # ZSCORE
3.0

As you can see, doubly added members get double scores.  This property is
what we will use in the middleware.

You can list values and these scores the sorted set contains.
Similar to :class:`dict` there's :meth:`~sider.sortedset.SortedSet.items()`
method.

>>> my_sorted_set.items()  # ZRANGE
[('https://bitbucket.org/dahlia/sider', 1.0),
 ('https://twitter.com/hongminhee', 2.0),
 ('http://dahlia.kr/', 2.0)]
>>> my_sorted_set.items(reverse=True)  # ZREVRANGE
[('http://dahlia.kr/', 2.0),
 ('https://twitter.com/hongminhee', 2.0),
 ('https://bitbucket.org/dahlia/sider', 1.0)]

There are other many features to :class:`~sider.sortedset.SortedSet` type,
but it's all we need to know to implement the middleware.  So we stop
introduction of the type to step forward.

Replace :class:`dict` with :class:`~sider.sortedset.SortedSet`
--------------------------------------------------------------

To replace :class:`dict` with :class:`~sider.sortedset.SortedSet`,
look :meth:`RefererStatMiddleware.__init__()` method first::

    def __init__(self, application):
        self.application = application
        self.referer_set = {}

.. note::

   The following codes implictly assumes that it imports::

       from redis.client import StrictClient
       from sider.session import Session
       from sider.types import SortedSet

The above code can be easily changed to:

::

    def __init__(self, application):
        assert callable(application)
        self.application = application
        client = StrictRedis()
        session = Session(client)
        self.referer_set = session.get('wsgi_referer_set', SortedSet)

It should be more configurable by users.  Redis key is currently hard-coded
as ``wsgi_referer_set``.  It can be parameterized, right? ::

    def __init__(self, set_key, application):
        assert callable(application)
        self.application = application
        client = StrictRedis()
        session = Session(client)
        self.referer_set = session.get(str(set_key), SortedSet)

It still lacks configurability.  Users can't set address of Redis server
to connect.  Parameterize ``session`` as well::

    def __init__(self, session, set_key, application):
        assert isinstance(session, Session)
        assert callable(application)
        self.application = application
        self.referer_set = session.get(str(set_key), SortedSet)

Okay, it's enough flexible to environments.  Our first and third problems
have just solved.  Its data become shared and don't be split anymore.
No data loss even if process has terminated.

Next, we have to make increment atomic.  See a part of
:meth:`RefererStatMiddleware.__call__()` method::

    try:
        self.referer_set[referer] += 1
    except KeyError:
        self.referer_set[referer] = 1

Redis sorted set offers a simple atomic way to increase its score:
:redis:`ZINCRBY`.  Sider maps :redis:`ZINCRBY` command to
:meth:`SortedSet.add() <sider.sortedset.SortedSet.add>` method.
So, those lines can be replaced by the following line::

    self.referer_set.add(referer)

and it will be committed atomically.

Source code
-----------

The complete source code of this example can be found in
:file:`examples/wsgi-referer-stat/` directory of the repository.

It's public domain, feel free!

Final API
---------

"""
from sider.session import Session
from sider.types import SortedSet
from jinja2 import Environment, PackageLoader

class RefererStatMiddleware(object):
    """A simple WSGI middleware that collects :mailheader:`Referer`
    headers and stores it into a Redis sorted set.

    You can see the list of referrers ordered by duplication count
    in ``/__stat__`` page (or you can configure the ``stat_path``
    argument).

    :param session: sider session object
    :type session: :class:`sider.session.Session`
    :param set_key: the key name of Redis sorted set
                    to store data
    :type set_key: :class:`basestring`
    :param application: wsgi app to wrap
    :type application: :class:`collections.Callable`
    :param stat_path: path to see the collected data.
                      default is ``'/__stat__'``.
                      if it's ``None`` the data cannot be accessed
                      from outside
    :type stat_path: :class:`basestring`

    """

    #: (:class:`sider.sortedset.SortedSet`) The set of collected
    #: :mailheader:`Referer` strings.
    referer_set = None

    def __init__(self, session, set_key, application, stat_path='/__stat__'):
        if not isinstance(session, Session):
            raise TypeError('session must be an instance of sider.session.'
                            'Session, not ' + repr(session))
        elif not callable(application):
            raise TypeError('application must be callable, not ' +
                            repr(application))
        self.session = session
        self.application = application
        self.referer_set = session.get(str(set_key), SortedSet)
        self.stat_path = stat_path
        loader = PackageLoader(__name__)
        environment = Environment(loader=loader)
        self.template = environment.get_template('stat.html')

    def stat_application(self, environ, start_response):
        """WSGI application that lists its collected referers."""
        content_type = 'text/html; charset=utf-8'
        start_response('200 OK', [('Content-Type', content_type)])
        referers = self.referer_set.items(reverse=True)
        stream = self.template.generate(referers=referers)
        for chunk in stream:
            yield chunk.encode('utf-8', 'ignore')

    def __call__(self, environ, start_response):
        if self.stat_path is not None:
            path = environ['PATH_INFO']
            if path == self.stat_path or path.startswith(self.stat_path + '/'):
                return self.stat_application(environ, start_response)
        try:
            referer = environ['HTTP_REFERER']
        except KeyError:
            pass
        else:
            self.referer_set.add(referer)
        return self.application(environ, start_response)

<|endoftext|>


<commit_before>#!/usr/bin/env python

# (C) Copyright IBM Corporation 2004, 2005
# All Rights Reserved.
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# on the rights to use, copy, modify, merge, publish, distribute, sub
# license, and/or sell copies of the Software, and to permit persons to whom
# the Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice (including the next
# paragraph) shall be included in all copies or substantial portions of the
# Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT.  IN NO EVENT SHALL
# IBM AND/OR ITS SUPPLIERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
# IN THE SOFTWARE.
#
# Authors:
#    Ian Romanick <idr@us.ibm.com>

import license
import gl_XML, glX_XML
import sys, getopt

class PrintGlProcs(gl_XML.gl_print_base):
    def __init__(self, es=False):
        gl_XML.gl_print_base.__init__(self)

        self.es = es
        self.name = "gl_procs.py (from Mesa)"
        self.license = license.bsd_license_template % ( \
"""Copyright (C) 1999-2001  Brian Paul   All Rights Reserved.
(C) Copyright IBM Corporation 2004, 2006""", "BRIAN PAUL, IBM")

    def printRealHeader(self):
        print """
/* This file is only included by glapi.c and is used for
 * the GetProcAddress() function
 */

typedef struct {
    GLint Name_offset;
#if defined(NEED_FUNCTION_POINTER) || defined(GLX_INDIRECT_RENDERING)
    _glapi_proc Address;
#endif
    GLuint Offset;
} glprocs_table_t;

#if   !defined(NEED_FUNCTION_POINTER) && !defined(GLX_INDIRECT_RENDERING)
#  define NAME_FUNC_OFFSET(n,f1,f2,f3,o) { n , o }
#elif  defined(NEED_FUNCTION_POINTER) && !defined(GLX_INDIRECT_RENDERING)
#  define NAME_FUNC_OFFSET(n,f1,f2,f3,o) { n , (_glapi_proc) f1 , o }
#elif  defined(NEED_FUNCTION_POINTER) &&  defined(GLX_INDIRECT_RENDERING)
#  define NAME_FUNC_OFFSET(n,f1,f2,f3,o) { n , (_glapi_proc) f2 , o }
#elif !defined(NEED_FUNCTION_POINTER) &&  defined(GLX_INDIRECT_RENDERING)
#  define NAME_FUNC_OFFSET(n,f1,f2,f3,o) { n , (_glapi_proc) f3 , o }
#endif

"""
        return

    def printRealFooter(self):
        print ''
        print '#undef NAME_FUNC_OFFSET'
        return

    def printFunctionString(self, name):
        print '    "gl%s\\0"' % (name)

    def printBody(self, api):
        print ''
        print 'static const char gl_string_table[] ='

        base_offset = 0
        table = []
        for func in api.functionIterateByOffset():
            name = func.dispatch_name()
            self.printFunctionString(func.name)
            table.append((base_offset, "gl" + name, "gl" + name, "NULL", func.offset))

            # The length of the function's name, plus 2 for "gl",
            # plus 1 for the NUL.

            base_offset += len(func.name) + 3

        for func in api.functionIterateByOffset():
            for n in func.entry_points:
                if n != func.name:
                    name = func.dispatch_name()
                    self.printFunctionString( n )

                    if func.has_different_protocol(n):
                        alt_name = "gl" + func.static_glx_name(n)
                        table.append((base_offset, "gl" + name, alt_name, alt_name, func.offset))
                    else:
                        table.append((base_offset, "gl" + name, "gl" + name, "NULL", func.offset))

                    base_offset += len(n) + 3

        print '    ;'
        print ''
        print ''
        print "#ifdef USE_MGL_NAMESPACE"
        for func in api.functionIterateByOffset():
            for n in func.entry_points:
                if (not func.is_static_entry_point(func.name)) or (func.has_different_protocol(n) and not func.is_static_entry_point(n)):
                    print '#define gl_dispatch_stub_%u mgl_dispatch_stub_%u' % (func.offset, func.offset)
                    break
        print "#endif /* USE_MGL_NAMESPACE */"
        print ''
        print ''
        print '#if defined(NEED_FUNCTION_POINTER) || defined(GLX_INDIRECT_RENDERING)'
        for func in api.functionIterateByOffset():
            for n in func.entry_points:
                if (not func.is_static_entry_point(func.name)) or (func.has_different_protocol(n) and not func.is_static_entry_point(n)):
                    print '%s GLAPIENTRY gl_dispatch_stub_%u(%s);' % (func.return_type, func.offset, func.get_parameter_string())
                    break

        if self.es:
            categories = {}
            for func in api.functionIterateByOffset():
                for n in func.entry_points:
                    cat, num = api.get_category_for_name(n)
                    if (cat.startswith("es") or cat.startswith("GL_OES")):
                        if not categories.has_key(cat):
                            categories[cat] = []
                        proto = 'GLAPI %s GLAPIENTRY %s(%s);' \
                                        % (func.return_type, "gl" + n, func.get_parameter_string(n))
                        categories[cat].append(proto)
            if categories:
                print ''
                print '/* OpenGL ES specific prototypes */'
                print ''
                keys = categories.keys()
                keys.sort()
                for key in keys:
                    print '/* category %s */' % key
                    print "\n".join(categories[key])
                print ''

        print '#endif /* defined(NEED_FUNCTION_POINTER) || defined(GLX_INDIRECT_RENDERING) */'

        print ''
        print 'static const glprocs_table_t static_functions[] = {'

        for info in table:
            print '    NAME_FUNC_OFFSET(%5u, %s, %s, %s, %d),' % info

        print '    NAME_FUNC_OFFSET(-1, NULL, NULL, NULL, 0)'
        print '};'
        return

def show_usage():
    print "Usage: %s [-f input_file_name] [-c]" % sys.argv[0]
    print "-c          Enable compatibility with OpenGL ES."
    sys.exit(1)

if __name__ == '__main__':
    file_name = "gl_API.xml"

    try:
        (args, trail) = getopt.getopt(sys.argv[1:], "f:c")
    except Exception,e:
        show_usage()

    es = False
    for (arg,val) in args:
        if arg == "-f":
            file_name = val
        elif arg == "-c":
            es = True

    api = gl_XML.parse_GL_API(file_name, glX_XML.glx_item_factory())
    printer = PrintGlProcs(es)
    printer.Print(api)
<commit_msg>glapi: gl_procs.py: Fix a few low hanging style things<commit_after>#!/usr/bin/env python

# (C) Copyright IBM Corporation 2004, 2005
# All Rights Reserved.
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# on the rights to use, copy, modify, merge, publish, distribute, sub
# license, and/or sell copies of the Software, and to permit persons to whom
# the Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice (including the next
# paragraph) shall be included in all copies or substantial portions of the
# Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT.  IN NO EVENT SHALL
# IBM AND/OR ITS SUPPLIERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
# IN THE SOFTWARE.
#
# Authors:
#    Ian Romanick <idr@us.ibm.com>

import sys
import getopt

import license
import gl_XML
import glX_XML

class PrintGlProcs(gl_XML.gl_print_base):
    def __init__(self, es=False):
        gl_XML.gl_print_base.__init__(self)

        self.es = es
        self.name = "gl_procs.py (from Mesa)"
        self.license = license.bsd_license_template % ( \
"""Copyright (C) 1999-2001  Brian Paul   All Rights Reserved.
(C) Copyright IBM Corporation 2004, 2006""", "BRIAN PAUL, IBM")

    def printRealHeader(self):
        print """
/* This file is only included by glapi.c and is used for
 * the GetProcAddress() function
 */

typedef struct {
    GLint Name_offset;
#if defined(NEED_FUNCTION_POINTER) || defined(GLX_INDIRECT_RENDERING)
    _glapi_proc Address;
#endif
    GLuint Offset;
} glprocs_table_t;

#if   !defined(NEED_FUNCTION_POINTER) && !defined(GLX_INDIRECT_RENDERING)
#  define NAME_FUNC_OFFSET(n,f1,f2,f3,o) { n , o }
#elif  defined(NEED_FUNCTION_POINTER) && !defined(GLX_INDIRECT_RENDERING)
#  define NAME_FUNC_OFFSET(n,f1,f2,f3,o) { n , (_glapi_proc) f1 , o }
#elif  defined(NEED_FUNCTION_POINTER) &&  defined(GLX_INDIRECT_RENDERING)
#  define NAME_FUNC_OFFSET(n,f1,f2,f3,o) { n , (_glapi_proc) f2 , o }
#elif !defined(NEED_FUNCTION_POINTER) &&  defined(GLX_INDIRECT_RENDERING)
#  define NAME_FUNC_OFFSET(n,f1,f2,f3,o) { n , (_glapi_proc) f3 , o }
#endif

"""
        return

    def printRealFooter(self):
        print ''
        print '#undef NAME_FUNC_OFFSET'
        return

    def printFunctionString(self, name):
        print '    "gl%s\\0"' % (name)

    def printBody(self, api):
        print ''
        print 'static const char gl_string_table[] ='

        base_offset = 0
        table = []
        for func in api.functionIterateByOffset():
            name = func.dispatch_name()
            self.printFunctionString(func.name)
            table.append((base_offset, "gl" + name, "gl" + name, "NULL", func.offset))

            # The length of the function's name, plus 2 for "gl",
            # plus 1 for the NUL.

            base_offset += len(func.name) + 3

        for func in api.functionIterateByOffset():
            for n in func.entry_points:
                if n != func.name:
                    name = func.dispatch_name()
                    self.printFunctionString( n )

                    if func.has_different_protocol(n):
                        alt_name = "gl" + func.static_glx_name(n)
                        table.append((base_offset, "gl" + name, alt_name, alt_name, func.offset))
                    else:
                        table.append((base_offset, "gl" + name, "gl" + name, "NULL", func.offset))

                    base_offset += len(n) + 3

        print '    ;'
        print ''
        print ''
        print "#ifdef USE_MGL_NAMESPACE"
        for func in api.functionIterateByOffset():
            for n in func.entry_points:
                if (not func.is_static_entry_point(func.name)) or (func.has_different_protocol(n) and not func.is_static_entry_point(n)):
                    print '#define gl_dispatch_stub_%u mgl_dispatch_stub_%u' % (func.offset, func.offset)
                    break
        print "#endif /* USE_MGL_NAMESPACE */"
        print ''
        print ''
        print '#if defined(NEED_FUNCTION_POINTER) || defined(GLX_INDIRECT_RENDERING)'
        for func in api.functionIterateByOffset():
            for n in func.entry_points:
                if (not func.is_static_entry_point(func.name)) or (func.has_different_protocol(n) and not func.is_static_entry_point(n)):
                    print '%s GLAPIENTRY gl_dispatch_stub_%u(%s);' % (func.return_type, func.offset, func.get_parameter_string())
                    break

        if self.es:
            categories = {}
            for func in api.functionIterateByOffset():
                for n in func.entry_points:
                    cat, num = api.get_category_for_name(n)
                    if (cat.startswith("es") or cat.startswith("GL_OES")):
                        if not categories.has_key(cat):
                            categories[cat] = []
                        proto = 'GLAPI %s GLAPIENTRY %s(%s);' \
                                        % (func.return_type, "gl" + n, func.get_parameter_string(n))
                        categories[cat].append(proto)
            if categories:
                print ''
                print '/* OpenGL ES specific prototypes */'
                print ''
                keys = categories.keys()
                keys.sort()
                for key in keys:
                    print '/* category %s */' % key
                    print "\n".join(categories[key])
                print ''

        print '#endif /* defined(NEED_FUNCTION_POINTER) || defined(GLX_INDIRECT_RENDERING) */'

        print ''
        print 'static const glprocs_table_t static_functions[] = {'

        for info in table:
            print '    NAME_FUNC_OFFSET(%5u, %s, %s, %s, %d),' % info

        print '    NAME_FUNC_OFFSET(-1, NULL, NULL, NULL, 0)'
        print '};'
        return

def show_usage():
    print "Usage: %s [-f input_file_name] [-c]" % sys.argv[0]
    print "-c          Enable compatibility with OpenGL ES."
    sys.exit(1)

def main():
    """Main function."""
    file_name = "gl_API.xml"

    try:
        args, _ = getopt.getopt(sys.argv[1:], "f:c")
    except Exception:
        show_usage()

    es = False
    for arg, val in args:
        if arg == "-f":
            file_name = val
        elif arg == "-c":
            es = True

    api = gl_XML.parse_GL_API(file_name, glX_XML.glx_item_factory())
    printer = PrintGlProcs(es)
    printer.Print(api)

if __name__ == '__main__':
    main()
<|endoftext|>


<commit_before>import urllib2

from util import headers
from checks import AgentCheck

class Lighttpd(AgentCheck):
    """Tracks basic connection/requests/workers metrics

    See http://redmine.lighttpd.net/projects/1/wiki/Docs_ModStatus for Lighttpd details
    See http://redmine.lighttpd.net/projects/lighttpd2/wiki/Mod_status for Lighttpd2 details
    """

    URL_SUFFIX_PER_VERSION = {
        1: '?auto',
        2: '?format=plain',
        'Unknown': '?auto'
    }

    GAUGES = {
        'IdleServers': 'lighttpd.performance.idle_server',
        'BusyServers': 'lighttpd.performance.busy_servers',
        'Uptime': 'lighttpd.performance.uptime',
        'Total kBytes': 'lighttpd.net.bytes',
        'Total Accesses': 'lighttpd.net.hits',
        'memory_usage': 'lighttpd2.performance.memory_usage',
        'requests_avg': 'lighttpd2.net.requests_avg',
        'traffic_out_avg': 'lighttpd2.net.bytes_out_avg',
        'traffic_in_avg': 'lighttpd2.net.bytes_in_avg',
        'connections_avg': 'lighttpd2.net.connections_avg',
        'connection_state_start': 'lighttpd2.connections.state_start',
        'connection_state_read_header': 'lighttpd2.connections.state_read_header',
        'connection_state_handle_request': 'lighttpd2.connections.state_handle_request',
        'connection_state_write_response': 'lighttpd2.connections.state_write_response',
        'connection_state_keep_alive': 'lighttpd2.connections.state_keep_alive',
        'requests_avg_5sec': 'lighttpd2.net.requests_avg_5sec',
        'traffic_out_avg_5sec': 'lighttpd2.net.bytes_out_avg_5sec',
        'traffic_in_avg_5sec': 'lighttpd2.net.bytes_in_avg_5sec',
        'connections_avg_5sec': 'lighttpd2.net.connections_avg_5sec',
    }

    COUNTERS = {
        'requests_abs': 'lighttpd2.net.requests_total',
        'traffic_out_abs': 'lighttpd2.net.bytes_out',
        'traffic_in_abs': 'lighttpd2.net.bytes_in',
        'connections_abs': 'lighttpd2.net.connections_total',
        'status_1xx': 'lighttpd2.response.status_1xx',
        'status_2xx': 'lighttpd2.response.status_2xx',
        'status_3xx': 'lighttpd2.response.status_3xx',
        'status_4xx': 'lighttpd2.response.status_4xx',
        'status_5xx': 'lighttpd2.response.status_5xx',
    }

    RATES = {
        'Total kBytes': 'lighttpd.net.bytes_per_s',
        'Total Accesses': 'lighttpd.net.request_per_s'
    }

    def __init__(self, name, init_config, agentConfig, instances=None):
        AgentCheck.__init__(self, name, init_config, agentConfig, instances)
        self.assumed_url = {}

    def check(self, instance):
        if 'lighttpd_status_url' not in instance:
            raise Exception("Missing 'lighttpd_status_url' variable in Lighttpd config")

        url = self.assumed_url.get(instance['lighttpd_status_url'], instance['lighttpd_status_url'])

        tags = instance.get('tags', [])
        self.log.debug("Connecting to %s" % url)
        req = urllib2.Request(url, None,
            headers(self.agentConfig))
        request = urllib2.urlopen(req)
        headers_resp = request.info().headers
        server_version = self._get_server_version(headers_resp)
        response = request.read()

        metric_count = 0
        # Loop through and extract the numerical values
        for line in response.split('\n'):
            values = line.split(': ')
            if len(values) == 2: # match
                metric, value = values
                try:
                    value = float(value)
                except ValueError:
                    continue

                # Special case: kBytes => bytes
                if metric == 'Total kBytes':
                    value = value * 1024

                # Send metric as a gauge, if applicable
                if metric in self.GAUGES:
                    metric_count += 1
                    metric_name = self.GAUGES[metric]
                    self.gauge(metric_name, value, tags=tags)

                # Send metric as a rate, if applicable
                if metric in self.RATES:
                    metric_count += 1
                    metric_name = self.RATES[metric]
                    self.rate(metric_name, value, tags=tags)

                # Send metric as a counter, if applicable
                if metric in self.COUNTERS:
                    metric_count += 1
                    metric_name = self.COUNTERS[metric]
                    self.increment(metric_name, value, tags=tags)

        if metric_count == 0:
            url_suffix = self.URL_SUFFIX_PER_VERSION[server_version]
            if self.assumed_url.get(instance['lighttpd_status_url'], None) is None and url[-len(url_suffix):] != url_suffix:
                self.assumed_url[instance['lighttpd_status_url']] = '%s%s' % (url, url_suffix)
                self.log.debug("Assuming url was not correct. Trying to add %s suffix to the url" % url_suffix)
                self.check(instance)
            else:
                raise Exception("No metrics were fetched for this instance. Make sure that %s is the proper url." % instance['lighttpd_status_url'])

    def _get_server_version(self, headers):
        for h in headers:
            if "Server:" not in h:
                continue
            try:
                version = int(h.split('/')[1][0])
            except Exception, e:
                self.log.debug("Error while trying to get server version %s" % str(e))
                version = "Unknown"
            self.log.debug("Lighttpd server version is %s" % version)
            return version

        self.log.debug("Lighttpd server version is Unknown")
        return "Unknown"

<commit_msg>Rename lighttpd2 metrics to lighttpd<commit_after>import urllib2

from util import headers
from checks import AgentCheck

class Lighttpd(AgentCheck):
    """Tracks basic connection/requests/workers metrics

    See http://redmine.lighttpd.net/projects/1/wiki/Docs_ModStatus for Lighttpd details
    See http://redmine.lighttpd.net/projects/lighttpd2/wiki/Mod_status for Lighttpd2 details
    """

    URL_SUFFIX_PER_VERSION = {
        1: '?auto',
        2: '?format=plain',
        'Unknown': '?auto'
    }

    GAUGES = {
        'IdleServers': 'lighttpd.performance.idle_server',
        'BusyServers': 'lighttpd.performance.busy_servers',
        'Uptime': 'lighttpd.performance.uptime',
        'Total kBytes': 'lighttpd.net.bytes',
        'Total Accesses': 'lighttpd.net.hits',
        'memory_usage': 'lighttpd.performance.memory_usage',
        'requests_avg': 'lighttpd.net.requests_avg',
        'traffic_out_avg': 'lighttpd.net.bytes_out_avg',
        'traffic_in_avg': 'lighttpd.net.bytes_in_avg',
        'connections_avg': 'lighttpd.net.connections_avg',
        'connection_state_start': 'lighttpd.connections.state_start',
        'connection_state_read_header': 'lighttpd.connections.state_read_header',
        'connection_state_handle_request': 'lighttpd.connections.state_handle_request',
        'connection_state_write_response': 'lighttpd.connections.state_write_response',
        'connection_state_keep_alive': 'lighttpd.connections.state_keep_alive',
        'requests_avg_5sec': 'lighttpd.net.requests_avg_5sec',
        'traffic_out_avg_5sec': 'lighttpd.net.bytes_out_avg_5sec',
        'traffic_in_avg_5sec': 'lighttpd.net.bytes_in_avg_5sec',
        'connections_avg_5sec': 'lighttpd.net.connections_avg_5sec',
    }

    COUNTERS = {
        'requests_abs': 'lighttpd.net.requests_total',
        'traffic_out_abs': 'lighttpd.net.bytes_out',
        'traffic_in_abs': 'lighttpd.net.bytes_in',
        'connections_abs': 'lighttpd.net.connections_total',
        'status_1xx': 'lighttpd.response.status_1xx',
        'status_2xx': 'lighttpd.response.status_2xx',
        'status_3xx': 'lighttpd.response.status_3xx',
        'status_4xx': 'lighttpd.response.status_4xx',
        'status_5xx': 'lighttpd.response.status_5xx',
    }

    RATES = {
        'Total kBytes': 'lighttpd.net.bytes_per_s',
        'Total Accesses': 'lighttpd.net.request_per_s'
    }

    def __init__(self, name, init_config, agentConfig, instances=None):
        AgentCheck.__init__(self, name, init_config, agentConfig, instances)
        self.assumed_url = {}

    def check(self, instance):
        if 'lighttpd_status_url' not in instance:
            raise Exception("Missing 'lighttpd_status_url' variable in Lighttpd config")

        url = self.assumed_url.get(instance['lighttpd_status_url'], instance['lighttpd_status_url'])

        tags = instance.get('tags', [])
        self.log.debug("Connecting to %s" % url)
        req = urllib2.Request(url, None,
            headers(self.agentConfig))
        request = urllib2.urlopen(req)
        headers_resp = request.info().headers
        server_version = self._get_server_version(headers_resp)
        response = request.read()

        metric_count = 0
        # Loop through and extract the numerical values
        for line in response.split('\n'):
            values = line.split(': ')
            if len(values) == 2: # match
                metric, value = values
                try:
                    value = float(value)
                except ValueError:
                    continue

                # Special case: kBytes => bytes
                if metric == 'Total kBytes':
                    value = value * 1024

                # Send metric as a gauge, if applicable
                if metric in self.GAUGES:
                    metric_count += 1
                    metric_name = self.GAUGES[metric]
                    self.gauge(metric_name, value, tags=tags)

                # Send metric as a rate, if applicable
                if metric in self.RATES:
                    metric_count += 1
                    metric_name = self.RATES[metric]
                    self.rate(metric_name, value, tags=tags)

                # Send metric as a counter, if applicable
                if metric in self.COUNTERS:
                    metric_count += 1
                    metric_name = self.COUNTERS[metric]
                    self.increment(metric_name, value, tags=tags)

        if metric_count == 0:
            url_suffix = self.URL_SUFFIX_PER_VERSION[server_version]
            if self.assumed_url.get(instance['lighttpd_status_url'], None) is None and url[-len(url_suffix):] != url_suffix:
                self.assumed_url[instance['lighttpd_status_url']] = '%s%s' % (url, url_suffix)
                self.log.debug("Assuming url was not correct. Trying to add %s suffix to the url" % url_suffix)
                self.check(instance)
            else:
                raise Exception("No metrics were fetched for this instance. Make sure that %s is the proper url." % instance['lighttpd_status_url'])

    def _get_server_version(self, headers):
        for h in headers:
            if "Server:" not in h:
                continue
            try:
                version = int(h.split('/')[1][0])
            except Exception, e:
                self.log.debug("Error while trying to get server version %s" % str(e))
                version = "Unknown"
            self.log.debug("Lighttpd server version is %s" % version)
            return version

        self.log.debug("Lighttpd server version is Unknown")
        return "Unknown"

<|endoftext|>


<commit_before>"""SCons.Tool.JavaCommon

Stuff for processing Java.

"""

#
# __COPYRIGHT__
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
# "Software"), to deal in the Software without restriction, including
# without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so, subject to
# the following conditions:
#
# The above copyright notice and this permission notice shall be included
# in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
# KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
#

__revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__"

import os
import os.path
import re

java_parsing = 1

default_java_version = '1.4'

if java_parsing:
    # Parse Java files for class names.
    #
    # This is a really cool parser from Charles Crain
    # that finds appropriate class names in Java source.

    # A regular expression that will find, in a java file:
    #     newlines;
    #     double-backslashes;
    #     a single-line comment "//";
    #     single or double quotes preceeded by a backslash;
    #     single quotes, double quotes, open or close braces, semi-colons,
    #         periods, open or close parentheses;
    #     floating-point numbers;
    #     any alphanumeric token (keyword, class name, specifier);
    #     any alphanumeric token surrounded by angle brackets (generics);
    #     the multi-line comment begin and end tokens /* and */;
    #     array declarations "[]".
    _reToken = re.compile(r'(\n|\\\\|//|\\[\'"]|[\'"\{\}\;\.\(\)]|' +
                          r'\d*\.\d*|[A-Za-z_][\w\$\.]*|<[A-Za-z_]\w+>|' +
                          r'/\*|\*/|\[\])')

    class OuterState:
        """The initial state for parsing a Java file for classes,
        interfaces, and anonymous inner classes."""
        def __init__(self, version=default_java_version):

            if not version in ('1.1', '1.2', '1.3','1.4', '1.5', '1.6',
                               '5', '6'):
                msg = "Java version %s not supported" % version
                raise NotImplementedError(msg)

            self.version = version
            self.listClasses = []
            self.listOutputs = []
            self.stackBrackets = []
            self.brackets = 0
            self.nextAnon = 1
            self.localClasses = []
            self.stackAnonClassBrackets = []
            self.anonStacksStack = [[0]]
            self.package = None

        def trace(self):
            pass

        def __getClassState(self):
            try:
                return self.classState
            except AttributeError:
                ret = ClassState(self)
                self.classState = ret
                return ret

        def __getPackageState(self):
            try:
                return self.packageState
            except AttributeError:
                ret = PackageState(self)
                self.packageState = ret
                return ret

        def __getAnonClassState(self):
            try:
                return self.anonState
            except AttributeError:
                self.outer_state = self
                ret = SkipState(1, AnonClassState(self))
                self.anonState = ret
                return ret

        def __getSkipState(self):
            try:
                return self.skipState
            except AttributeError:
                ret = SkipState(1, self)
                self.skipState = ret
                return ret

        def __getAnonStack(self):
            return self.anonStacksStack[-1]

        def openBracket(self):
            self.brackets = self.brackets + 1

        def closeBracket(self):
            self.brackets = self.brackets - 1
            if len(self.stackBrackets) and \
               self.brackets == self.stackBrackets[-1]:
                self.listOutputs.append('$'.join(self.listClasses))
                self.localClasses.pop()
                self.listClasses.pop()
                self.anonStacksStack.pop()
                self.stackBrackets.pop()
            if len(self.stackAnonClassBrackets) and \
               self.brackets == self.stackAnonClassBrackets[-1]:
                self.__getAnonStack().pop()
                self.stackAnonClassBrackets.pop()

        def parseToken(self, token):
            if token[:2] == '//':
                return IgnoreState('\n', self)
            elif token == '/*':
                return IgnoreState('*/', self)
            elif token == '{':
                self.openBracket()
            elif token == '}':
                self.closeBracket()
            elif token in [ '"', "'" ]:
                return IgnoreState(token, self)
            elif token == "new":
                # anonymous inner class
                if len(self.listClasses) > 0:
                    return self.__getAnonClassState()
                return self.__getSkipState() # Skip the class name
            elif token in ['class', 'interface', 'enum']:
                if len(self.listClasses) == 0:
                    self.nextAnon = 1
                self.stackBrackets.append(self.brackets)
                return self.__getClassState()
            elif token == 'package':
                return self.__getPackageState()
            elif token == '.':
                # Skip the attribute, it might be named "class", in which
                # case we don't want to treat the following token as
                # an inner class name...
                return self.__getSkipState()
            return self

        def addAnonClass(self):
            """Add an anonymous inner class"""
            if self.version in ('1.1', '1.2', '1.3', '1.4'):
                clazz = self.listClasses[0]
                self.listOutputs.append('%s$%d' % (clazz, self.nextAnon))
            elif self.version in ('1.5', '1.6', '5', '6'):
                self.stackAnonClassBrackets.append(self.brackets)
                className = []
                className.extend(self.listClasses)
                self.__getAnonStack()[-1] = self.__getAnonStack()[-1] + 1
                for anon in self.__getAnonStack():
                    className.append(str(anon))
                self.listOutputs.append('$'.join(className))

            self.nextAnon = self.nextAnon + 1
            self.__getAnonStack().append(0)

        def setPackage(self, package):
            self.package = package

    class AnonClassState:
        """A state that looks for anonymous inner classes."""
        def __init__(self, old_state):
            # outer_state is always an instance of OuterState
            self.outer_state = old_state.outer_state
            self.old_state = old_state
            self.brace_level = 0
        def parseToken(self, token):
            # This is an anonymous class if and only if the next
            # non-whitespace token is a bracket. Everything between
            # braces should be parsed as normal java code.
            if token[:2] == '//':
                return IgnoreState('\n', self)
            elif token == '/*':
                return IgnoreState('*/', self)
            elif token == '\n':
                return self
            elif token[0] == '<' and token[-1] == '>':
                return self
            elif token == '(':
                self.brace_level = self.brace_level + 1
                return self
            if self.brace_level > 0:
                if token == 'new':
                    # look further for anonymous inner class
                    return SkipState(1, AnonClassState(self))
                elif token in [ '"', "'" ]:
                    return IgnoreState(token, self)
                elif token == ')':
                    self.brace_level = self.brace_level - 1
                return self
            if token == '{':
                self.outer_state.addAnonClass()
            return self.old_state.parseToken(token)

    class SkipState:
        """A state that will skip a specified number of tokens before
        reverting to the previous state."""
        def __init__(self, tokens_to_skip, old_state):
            self.tokens_to_skip = tokens_to_skip
            self.old_state = old_state
        def parseToken(self, token):
            self.tokens_to_skip = self.tokens_to_skip - 1
            if self.tokens_to_skip < 1:
                return self.old_state
            return self

    class ClassState:
        """A state we go into when we hit a class or interface keyword."""
        def __init__(self, outer_state):
            # outer_state is always an instance of OuterState
            self.outer_state = outer_state
        def parseToken(self, token):
            # the next non-whitespace token should be the name of the class
            if token == '\n':
                return self
            # If that's an inner class which is declared in a method, it
            # requires an index prepended to the class-name, e.g.
            # 'Foo$1Inner' (Tigris Issue 2087)
            if self.outer_state.localClasses and \
                self.outer_state.stackBrackets[-1] > \
                self.outer_state.stackBrackets[-2]+1:
                locals = self.outer_state.localClasses[-1]
                try:
                    idx = locals[token]
                    locals[token] = locals[token]+1
                except KeyError:
                    locals[token] = 1
                token = str(locals[token]) + token
            self.outer_state.localClasses.append({})
            self.outer_state.listClasses.append(token)
            self.outer_state.anonStacksStack.append([0])
            return self.outer_state

    class IgnoreState:
        """A state that will ignore all tokens until it gets to a
        specified token."""
        def __init__(self, ignore_until, old_state):
            self.ignore_until = ignore_until
            self.old_state = old_state
        def parseToken(self, token):
            if self.ignore_until == token:
                return self.old_state
            return self

    class PackageState:
        """The state we enter when we encounter the package keyword.
        We assume the next token will be the package name."""
        def __init__(self, outer_state):
            # outer_state is always an instance of OuterState
            self.outer_state = outer_state
        def parseToken(self, token):
            self.outer_state.setPackage(token)
            return self.outer_state

    def parse_java_file(fn, version=default_java_version):
        return parse_java(open(fn, 'r').read(), version)

    def parse_java(contents, version=default_java_version, trace=None):
        """Parse a .java file and return a double of package directory,
        plus a list of .class files that compiling that .java file will
        produce"""
        package = None
        initial = OuterState(version)
        currstate = initial
        for token in _reToken.findall(contents):
            # The regex produces a bunch of groups, but only one will
            # have anything in it.
            currstate = currstate.parseToken(token)
            if trace: trace(token, currstate)
        if initial.package:
            package = initial.package.replace('.', os.sep)
        return (package, initial.listOutputs)

else:
    # Don't actually parse Java files for class names.
    #
    # We might make this a configurable option in the future if
    # Java-file parsing takes too long (although it shouldn't relative
    # to how long the Java compiler itself seems to take...).

    def parse_java_file(fn):
        """ "Parse" a .java file.

        This actually just splits the file name, so the assumption here
        is that the file name matches the public class name, and that
        the path to the file is the same as the package name.
        """
        return os.path.split(file)

# Local Variables:
# tab-width:4
# indent-tabs-mode:nil
# End:
# vim: set expandtab tabstop=4 shiftwidth=4:
<commit_msg>Convert old-style classes in tool/JavaCommon.py to new-style classes.<commit_after>"""SCons.Tool.JavaCommon

Stuff for processing Java.

"""

#
# __COPYRIGHT__
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
# "Software"), to deal in the Software without restriction, including
# without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so, subject to
# the following conditions:
#
# The above copyright notice and this permission notice shall be included
# in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
# KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
#

__revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__"

import os
import os.path
import re

java_parsing = 1

default_java_version = '1.4'

if java_parsing:
    # Parse Java files for class names.
    #
    # This is a really cool parser from Charles Crain
    # that finds appropriate class names in Java source.

    # A regular expression that will find, in a java file:
    #     newlines;
    #     double-backslashes;
    #     a single-line comment "//";
    #     single or double quotes preceeded by a backslash;
    #     single quotes, double quotes, open or close braces, semi-colons,
    #         periods, open or close parentheses;
    #     floating-point numbers;
    #     any alphanumeric token (keyword, class name, specifier);
    #     any alphanumeric token surrounded by angle brackets (generics);
    #     the multi-line comment begin and end tokens /* and */;
    #     array declarations "[]".
    _reToken = re.compile(r'(\n|\\\\|//|\\[\'"]|[\'"\{\}\;\.\(\)]|' +
                          r'\d*\.\d*|[A-Za-z_][\w\$\.]*|<[A-Za-z_]\w+>|' +
                          r'/\*|\*/|\[\])')

    class OuterState(object):
        """The initial state for parsing a Java file for classes,
        interfaces, and anonymous inner classes."""
        def __init__(self, version=default_java_version):

            if not version in ('1.1', '1.2', '1.3','1.4', '1.5', '1.6',
                               '5', '6'):
                msg = "Java version %s not supported" % version
                raise NotImplementedError(msg)

            self.version = version
            self.listClasses = []
            self.listOutputs = []
            self.stackBrackets = []
            self.brackets = 0
            self.nextAnon = 1
            self.localClasses = []
            self.stackAnonClassBrackets = []
            self.anonStacksStack = [[0]]
            self.package = None

        def trace(self):
            pass

        def __getClassState(self):
            try:
                return self.classState
            except AttributeError:
                ret = ClassState(self)
                self.classState = ret
                return ret

        def __getPackageState(self):
            try:
                return self.packageState
            except AttributeError:
                ret = PackageState(self)
                self.packageState = ret
                return ret

        def __getAnonClassState(self):
            try:
                return self.anonState
            except AttributeError:
                self.outer_state = self
                ret = SkipState(1, AnonClassState(self))
                self.anonState = ret
                return ret

        def __getSkipState(self):
            try:
                return self.skipState
            except AttributeError:
                ret = SkipState(1, self)
                self.skipState = ret
                return ret

        def __getAnonStack(self):
            return self.anonStacksStack[-1]

        def openBracket(self):
            self.brackets = self.brackets + 1

        def closeBracket(self):
            self.brackets = self.brackets - 1
            if len(self.stackBrackets) and \
               self.brackets == self.stackBrackets[-1]:
                self.listOutputs.append('$'.join(self.listClasses))
                self.localClasses.pop()
                self.listClasses.pop()
                self.anonStacksStack.pop()
                self.stackBrackets.pop()
            if len(self.stackAnonClassBrackets) and \
               self.brackets == self.stackAnonClassBrackets[-1]:
                self.__getAnonStack().pop()
                self.stackAnonClassBrackets.pop()

        def parseToken(self, token):
            if token[:2] == '//':
                return IgnoreState('\n', self)
            elif token == '/*':
                return IgnoreState('*/', self)
            elif token == '{':
                self.openBracket()
            elif token == '}':
                self.closeBracket()
            elif token in [ '"', "'" ]:
                return IgnoreState(token, self)
            elif token == "new":
                # anonymous inner class
                if len(self.listClasses) > 0:
                    return self.__getAnonClassState()
                return self.__getSkipState() # Skip the class name
            elif token in ['class', 'interface', 'enum']:
                if len(self.listClasses) == 0:
                    self.nextAnon = 1
                self.stackBrackets.append(self.brackets)
                return self.__getClassState()
            elif token == 'package':
                return self.__getPackageState()
            elif token == '.':
                # Skip the attribute, it might be named "class", in which
                # case we don't want to treat the following token as
                # an inner class name...
                return self.__getSkipState()
            return self

        def addAnonClass(self):
            """Add an anonymous inner class"""
            if self.version in ('1.1', '1.2', '1.3', '1.4'):
                clazz = self.listClasses[0]
                self.listOutputs.append('%s$%d' % (clazz, self.nextAnon))
            elif self.version in ('1.5', '1.6', '5', '6'):
                self.stackAnonClassBrackets.append(self.brackets)
                className = []
                className.extend(self.listClasses)
                self.__getAnonStack()[-1] = self.__getAnonStack()[-1] + 1
                for anon in self.__getAnonStack():
                    className.append(str(anon))
                self.listOutputs.append('$'.join(className))

            self.nextAnon = self.nextAnon + 1
            self.__getAnonStack().append(0)

        def setPackage(self, package):
            self.package = package

    class AnonClassState(object):
        """A state that looks for anonymous inner classes."""
        def __init__(self, old_state):
            # outer_state is always an instance of OuterState
            self.outer_state = old_state.outer_state
            self.old_state = old_state
            self.brace_level = 0
        def parseToken(self, token):
            # This is an anonymous class if and only if the next
            # non-whitespace token is a bracket. Everything between
            # braces should be parsed as normal java code.
            if token[:2] == '//':
                return IgnoreState('\n', self)
            elif token == '/*':
                return IgnoreState('*/', self)
            elif token == '\n':
                return self
            elif token[0] == '<' and token[-1] == '>':
                return self
            elif token == '(':
                self.brace_level = self.brace_level + 1
                return self
            if self.brace_level > 0:
                if token == 'new':
                    # look further for anonymous inner class
                    return SkipState(1, AnonClassState(self))
                elif token in [ '"', "'" ]:
                    return IgnoreState(token, self)
                elif token == ')':
                    self.brace_level = self.brace_level - 1
                return self
            if token == '{':
                self.outer_state.addAnonClass()
            return self.old_state.parseToken(token)

    class SkipState(object):
        """A state that will skip a specified number of tokens before
        reverting to the previous state."""
        def __init__(self, tokens_to_skip, old_state):
            self.tokens_to_skip = tokens_to_skip
            self.old_state = old_state
        def parseToken(self, token):
            self.tokens_to_skip = self.tokens_to_skip - 1
            if self.tokens_to_skip < 1:
                return self.old_state
            return self

    class ClassState(object):
        """A state we go into when we hit a class or interface keyword."""
        def __init__(self, outer_state):
            # outer_state is always an instance of OuterState
            self.outer_state = outer_state
        def parseToken(self, token):
            # the next non-whitespace token should be the name of the class
            if token == '\n':
                return self
            # If that's an inner class which is declared in a method, it
            # requires an index prepended to the class-name, e.g.
            # 'Foo$1Inner' (Tigris Issue 2087)
            if self.outer_state.localClasses and \
                self.outer_state.stackBrackets[-1] > \
                self.outer_state.stackBrackets[-2]+1:
                locals = self.outer_state.localClasses[-1]
                try:
                    idx = locals[token]
                    locals[token] = locals[token]+1
                except KeyError:
                    locals[token] = 1
                token = str(locals[token]) + token
            self.outer_state.localClasses.append({})
            self.outer_state.listClasses.append(token)
            self.outer_state.anonStacksStack.append([0])
            return self.outer_state

    class IgnoreState(object):
        """A state that will ignore all tokens until it gets to a
        specified token."""
        def __init__(self, ignore_until, old_state):
            self.ignore_until = ignore_until
            self.old_state = old_state
        def parseToken(self, token):
            if self.ignore_until == token:
                return self.old_state
            return self

    class PackageState(object):
        """The state we enter when we encounter the package keyword.
        We assume the next token will be the package name."""
        def __init__(self, outer_state):
            # outer_state is always an instance of OuterState
            self.outer_state = outer_state
        def parseToken(self, token):
            self.outer_state.setPackage(token)
            return self.outer_state

    def parse_java_file(fn, version=default_java_version):
        return parse_java(open(fn, 'r').read(), version)

    def parse_java(contents, version=default_java_version, trace=None):
        """Parse a .java file and return a double of package directory,
        plus a list of .class files that compiling that .java file will
        produce"""
        package = None
        initial = OuterState(version)
        currstate = initial
        for token in _reToken.findall(contents):
            # The regex produces a bunch of groups, but only one will
            # have anything in it.
            currstate = currstate.parseToken(token)
            if trace: trace(token, currstate)
        if initial.package:
            package = initial.package.replace('.', os.sep)
        return (package, initial.listOutputs)

else:
    # Don't actually parse Java files for class names.
    #
    # We might make this a configurable option in the future if
    # Java-file parsing takes too long (although it shouldn't relative
    # to how long the Java compiler itself seems to take...).

    def parse_java_file(fn):
        """ "Parse" a .java file.

        This actually just splits the file name, so the assumption here
        is that the file name matches the public class name, and that
        the path to the file is the same as the package name.
        """
        return os.path.split(file)

# Local Variables:
# tab-width:4
# indent-tabs-mode:nil
# End:
# vim: set expandtab tabstop=4 shiftwidth=4:
<|endoftext|>


<commit_before>#
# __COPYRIGHT__
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
# "Software"), to deal in the Software without restriction, including
# without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so, subject to
# the following conditions:
#
# The above copyright notice and this permission notice shall be included
# in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
# KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
#

__doc__ = """
SCons compatibility package for old Python versions

This subpackage holds modules that provide backwards-compatible
implementations of various things that we'd like to use in SCons but which
only show up in later versions of Python than the early, old version(s)
we still support.

Other code will not generally reference things in this package through
the SCons.compat namespace.  The modules included here add things to
the __builtin__ namespace or the global module list so that the rest
of our code can use the objects and names imported here regardless of
Python version.

Simply enough, things that go in the __builtin__ name space come from
our builtins module.

The rest of the things here will be in individual compatibility modules
that are either: 1) suitably modified copies of the future modules that
we want to use; or 2) backwards compatible re-implementations of the
specific portions of a future module's API that we want to use.

GENERAL WARNINGS:  Implementations of functions in the SCons.compat
modules are *NOT* guaranteed to be fully compliant with these functions in
later versions of Python.  We are only concerned with adding functionality
that we actually use in SCons, so be wary if you lift this code for
other uses.  (That said, making these more nearly the same as later,
official versions is still a desirable goal, we just don't need to be
obsessive about it.)

We name the compatibility modules with an initial '_scons_' (for example,
_scons_subprocess.py is our compatibility module for subprocess) so
that we can still try to import the real module name and fall back to
our compatibility module if we get an ImportError.  The import_as()
function defined below loads the module as the "real" name (without the
'_scons'), after which all of the "import {module}" statements in the
rest of our code will find our pre-loaded compatibility module.
"""

__revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__"

def import_as(module, name):
    """
    Imports the specified module (from our local directory) as the
    specified name.
    """
    import imp
    import os.path
    dir = os.path.split(__file__)[0]
    file, filename, suffix_mode_type = imp.find_module(module, [dir])
    imp.load_module(name, file, filename, suffix_mode_type)

import builtins

try:
    import hashlib
except ImportError:
    # Pre-2.5 Python has no hashlib module.
    try:
        import_as('_scons_hashlib', 'hashlib')
    except ImportError:
        # If we failed importing our compatibility module, it probably
        # means this version of Python has no md5 module.  Don't do
        # anything and let the higher layer discover this fact, so it
        # can fall back to using timestamp.
        pass

try:
    set
except NameError:
    # Pre-2.4 Python has no native set type
    try:
        # Python 2.2 and 2.3 can use the copy of the 2.[45] sets module
        # that we grabbed.
        import_as('_scons_sets', 'sets')
    except (ImportError, SyntaxError):
        # Python 1.5 (ImportError, no __future_ module) and 2.1
        # (SyntaxError, no generators in __future__) will blow up
        # trying to import the 2.[45] sets module, so back off to a
        # custom sets module that can be discarded easily when we
        # stop supporting those versions.
        import_as('_scons_sets15', 'sets')
    import __builtin__
    import sets
    __builtin__.set = sets.Set

import fnmatch
try:
    fnmatch.filter
except AttributeError:
    # Pre-2.2 Python has no fnmatch.filter() function.
    def filter(names, pat):
        """Return the subset of the list NAMES that match PAT"""
        import os,posixpath
        result=[]
        pat = os.path.normcase(pat)
        if not fnmatch._cache.has_key(pat):
            import re
            res = fnmatch.translate(pat)
            fnmatch._cache[pat] = re.compile(res)
        match = fnmatch._cache[pat].match
        if os.path is posixpath:
            # normcase on posix is NOP. Optimize it away from the loop.
            for name in names:
                if match(name):
                    result.append(name)
        else:
            for name in names:
                if match(os.path.normcase(name)):
                    result.append(name)
        return result
    fnmatch.filter = filter
    del filter

try:
    import itertools
except ImportError:
    # Pre-2.3 Python has no itertools module.
    import_as('_scons_itertools', 'itertools')

# If we need the compatibility version of textwrap, it  must be imported
# before optparse, which uses it.
try:
    import textwrap
except ImportError:
    # Pre-2.3 Python has no textwrap module.
    import_as('_scons_textwrap', 'textwrap')

try:
    import optparse
except ImportError:
    # Pre-2.3 Python has no optparse module.
    import_as('_scons_optparse', 'optparse')

import os
try:
    os.devnull
except AttributeError:
    # Pre-2.4 Python has no os.devnull attribute
    import sys
    _names = sys.builtin_module_names
    if 'posix' in _names:
        os.devnull = '/dev/null'
    elif 'nt' in _names:
        os.devnull = 'nul'
    os.path.devnull = os.devnull
try:
    os.path.lexists
except AttributeError:
    # Pre-2.4 Python has no os.path.lexists function
    def lexists(path):
        return os.path.exists(path) or os.path.islink(path)
    os.path.lexists = lexists

import shlex
try:
    shlex.split
except AttributeError:
    # Pre-2.3 Python has no shlex.split() function.
    #
    # The full white-space splitting semantics of shlex.split() are
    # complicated to reproduce by hand, so just use a compatibility
    # version of the shlex module cribbed from Python 2.5 with some
    # minor modifications for older Python versions.
    del shlex
    import_as('_scons_shlex', 'shlex')

import shutil
try:
    shutil.move
except AttributeError:
    # Pre-2.3 Python has no shutil.move() function.
    #
    # Cribbed from Python 2.5.
    import os

    def move(src, dst):
        """Recursively move a file or directory to another location.

        If the destination is on our current filesystem, then simply use
        rename.  Otherwise, copy src to the dst and then remove src.
        A lot more could be done here...  A look at a mv.c shows a lot of
        the issues this implementation glosses over.

        """
        try:
            os.rename(src, dst)
        except OSError:
            if os.path.isdir(src):
                if shutil.destinsrc(src, dst):
                    raise Error, "Cannot move a directory '%s' into itself '%s'." % (src, dst)
                shutil.copytree(src, dst, symlinks=True)
                shutil.rmtree(src)
            else:
                shutil.copy2(src,dst)
                os.unlink(src)
    shutil.move = move
    del move

    def destinsrc(src, dst):
        src = os.path.abspath(src)
        return os.path.abspath(dst)[:len(src)] == src
    shutil.destinsrc = destinsrc
    del destinsrc

try:
    import subprocess
except ImportError:
    # Pre-2.4 Python has no subprocess module.
    import_as('_scons_subprocess', 'subprocess')

import sys
try:
    sys.version_info
except AttributeError:
    # Pre-1.6 Python has no sys.version_info
    import string
    version_string = string.split(sys.version)[0]
    version_ints = map(int, string.split(version_string, '.'))
    sys.version_info = tuple(version_ints + ['final', 0])

try:
    import UserString
except ImportError:
    # Pre-1.6 Python has no UserString module.
    import_as('_scons_UserString', 'UserString')

import tempfile
try:
    tempfile.mkstemp
except AttributeError:
    # Pre-2.3 Python has no tempfile.mkstemp function, so try to simulate it.
    # adapted from the mkstemp implementation in python 3.
    import os
    import errno
    def mkstemp( *args, **kw ) :
        text = False
        if 'text' in kw :
            text = kw['text']
            del kw['text']
        elif len( args ) == 4 :
            text = args[3]
            args = args[:3]
        flags = os.O_RDWR | os.O_CREAT | os.O_EXCL
        if not text and hasattr( os, 'O_BINARY' ) :
            flags = flags | os.O_BINARY
        while True:
            try :
                name = tempfile.mktemp( *args, **kw )
                fd = os.open( name, flags, 0600 )
                return (fd, os.path.abspath(name))
            except OSError, e:
                if e.errno == errno.EEXIST:
                    continue
                raise

    tempfile.mkstemp = mkstemp
    del mkstemp

# Local Variables:
# tab-width:4
# indent-tabs-mode:nil
# End:
# vim: set expandtab tabstop=4 shiftwidth=4:
<commit_msg>1.5 fix:  use apply() instead of (*args, **kw).<commit_after>#
# __COPYRIGHT__
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
# "Software"), to deal in the Software without restriction, including
# without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so, subject to
# the following conditions:
#
# The above copyright notice and this permission notice shall be included
# in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
# KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
#

__doc__ = """
SCons compatibility package for old Python versions

This subpackage holds modules that provide backwards-compatible
implementations of various things that we'd like to use in SCons but which
only show up in later versions of Python than the early, old version(s)
we still support.

Other code will not generally reference things in this package through
the SCons.compat namespace.  The modules included here add things to
the __builtin__ namespace or the global module list so that the rest
of our code can use the objects and names imported here regardless of
Python version.

Simply enough, things that go in the __builtin__ name space come from
our builtins module.

The rest of the things here will be in individual compatibility modules
that are either: 1) suitably modified copies of the future modules that
we want to use; or 2) backwards compatible re-implementations of the
specific portions of a future module's API that we want to use.

GENERAL WARNINGS:  Implementations of functions in the SCons.compat
modules are *NOT* guaranteed to be fully compliant with these functions in
later versions of Python.  We are only concerned with adding functionality
that we actually use in SCons, so be wary if you lift this code for
other uses.  (That said, making these more nearly the same as later,
official versions is still a desirable goal, we just don't need to be
obsessive about it.)

We name the compatibility modules with an initial '_scons_' (for example,
_scons_subprocess.py is our compatibility module for subprocess) so
that we can still try to import the real module name and fall back to
our compatibility module if we get an ImportError.  The import_as()
function defined below loads the module as the "real" name (without the
'_scons'), after which all of the "import {module}" statements in the
rest of our code will find our pre-loaded compatibility module.
"""

__revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__"

def import_as(module, name):
    """
    Imports the specified module (from our local directory) as the
    specified name.
    """
    import imp
    import os.path
    dir = os.path.split(__file__)[0]
    file, filename, suffix_mode_type = imp.find_module(module, [dir])
    imp.load_module(name, file, filename, suffix_mode_type)

import builtins

try:
    import hashlib
except ImportError:
    # Pre-2.5 Python has no hashlib module.
    try:
        import_as('_scons_hashlib', 'hashlib')
    except ImportError:
        # If we failed importing our compatibility module, it probably
        # means this version of Python has no md5 module.  Don't do
        # anything and let the higher layer discover this fact, so it
        # can fall back to using timestamp.
        pass

try:
    set
except NameError:
    # Pre-2.4 Python has no native set type
    try:
        # Python 2.2 and 2.3 can use the copy of the 2.[45] sets module
        # that we grabbed.
        import_as('_scons_sets', 'sets')
    except (ImportError, SyntaxError):
        # Python 1.5 (ImportError, no __future_ module) and 2.1
        # (SyntaxError, no generators in __future__) will blow up
        # trying to import the 2.[45] sets module, so back off to a
        # custom sets module that can be discarded easily when we
        # stop supporting those versions.
        import_as('_scons_sets15', 'sets')
    import __builtin__
    import sets
    __builtin__.set = sets.Set

import fnmatch
try:
    fnmatch.filter
except AttributeError:
    # Pre-2.2 Python has no fnmatch.filter() function.
    def filter(names, pat):
        """Return the subset of the list NAMES that match PAT"""
        import os,posixpath
        result=[]
        pat = os.path.normcase(pat)
        if not fnmatch._cache.has_key(pat):
            import re
            res = fnmatch.translate(pat)
            fnmatch._cache[pat] = re.compile(res)
        match = fnmatch._cache[pat].match
        if os.path is posixpath:
            # normcase on posix is NOP. Optimize it away from the loop.
            for name in names:
                if match(name):
                    result.append(name)
        else:
            for name in names:
                if match(os.path.normcase(name)):
                    result.append(name)
        return result
    fnmatch.filter = filter
    del filter

try:
    import itertools
except ImportError:
    # Pre-2.3 Python has no itertools module.
    import_as('_scons_itertools', 'itertools')

# If we need the compatibility version of textwrap, it  must be imported
# before optparse, which uses it.
try:
    import textwrap
except ImportError:
    # Pre-2.3 Python has no textwrap module.
    import_as('_scons_textwrap', 'textwrap')

try:
    import optparse
except ImportError:
    # Pre-2.3 Python has no optparse module.
    import_as('_scons_optparse', 'optparse')

import os
try:
    os.devnull
except AttributeError:
    # Pre-2.4 Python has no os.devnull attribute
    import sys
    _names = sys.builtin_module_names
    if 'posix' in _names:
        os.devnull = '/dev/null'
    elif 'nt' in _names:
        os.devnull = 'nul'
    os.path.devnull = os.devnull
try:
    os.path.lexists
except AttributeError:
    # Pre-2.4 Python has no os.path.lexists function
    def lexists(path):
        return os.path.exists(path) or os.path.islink(path)
    os.path.lexists = lexists

import shlex
try:
    shlex.split
except AttributeError:
    # Pre-2.3 Python has no shlex.split() function.
    #
    # The full white-space splitting semantics of shlex.split() are
    # complicated to reproduce by hand, so just use a compatibility
    # version of the shlex module cribbed from Python 2.5 with some
    # minor modifications for older Python versions.
    del shlex
    import_as('_scons_shlex', 'shlex')

import shutil
try:
    shutil.move
except AttributeError:
    # Pre-2.3 Python has no shutil.move() function.
    #
    # Cribbed from Python 2.5.
    import os

    def move(src, dst):
        """Recursively move a file or directory to another location.

        If the destination is on our current filesystem, then simply use
        rename.  Otherwise, copy src to the dst and then remove src.
        A lot more could be done here...  A look at a mv.c shows a lot of
        the issues this implementation glosses over.

        """
        try:
            os.rename(src, dst)
        except OSError:
            if os.path.isdir(src):
                if shutil.destinsrc(src, dst):
                    raise Error, "Cannot move a directory '%s' into itself '%s'." % (src, dst)
                shutil.copytree(src, dst, symlinks=True)
                shutil.rmtree(src)
            else:
                shutil.copy2(src,dst)
                os.unlink(src)
    shutil.move = move
    del move

    def destinsrc(src, dst):
        src = os.path.abspath(src)
        return os.path.abspath(dst)[:len(src)] == src
    shutil.destinsrc = destinsrc
    del destinsrc

try:
    import subprocess
except ImportError:
    # Pre-2.4 Python has no subprocess module.
    import_as('_scons_subprocess', 'subprocess')

import sys
try:
    sys.version_info
except AttributeError:
    # Pre-1.6 Python has no sys.version_info
    import string
    version_string = string.split(sys.version)[0]
    version_ints = map(int, string.split(version_string, '.'))
    sys.version_info = tuple(version_ints + ['final', 0])

try:
    import UserString
except ImportError:
    # Pre-1.6 Python has no UserString module.
    import_as('_scons_UserString', 'UserString')

import tempfile
try:
    tempfile.mkstemp
except AttributeError:
    # Pre-2.3 Python has no tempfile.mkstemp function, so try to simulate it.
    # adapted from the mkstemp implementation in python 3.
    import os
    import errno
    def mkstemp(*args, **kw):
        text = False
        if 'text' in kw :
            text = kw['text']
            del kw['text']
        elif len( args ) == 4 :
            text = args[3]
            args = args[:3]
        flags = os.O_RDWR | os.O_CREAT | os.O_EXCL
        if not text and hasattr( os, 'O_BINARY' ) :
            flags = flags | os.O_BINARY
        while True:
            try :
                name = apply(tempfile.mktemp, args, kw)
                fd = os.open( name, flags, 0600 )
                return (fd, os.path.abspath(name))
            except OSError, e:
                if e.errno == errno.EEXIST:
                    continue
                raise

    tempfile.mkstemp = mkstemp
    del mkstemp

# Local Variables:
# tab-width:4
# indent-tabs-mode:nil
# End:
# vim: set expandtab tabstop=4 shiftwidth=4:
<|endoftext|>