<commit_before>"""
This module is intended to document the main interface with the OpenID
consumer libary.  The only part of the library which has to be used
and isn't documented in full here is the store required to create an
C{OpenIDConsumer instance}.  More on the abstract store type and
concrete implementations of it that are provided in the documentation
for the C{__init__} method of the C{OpenIDConsumer} class.

OVERVIEW
========

    The OpenID identity verification process most commonly uses the
    following steps, as visible to the user of this library:

    1. The user enters their OpenID into a field on the consumer's
       site, and hits a log in button.

    2. The consumer site checks that the entered URL describes an
       OpenID page by fetching it and looking for appropriate
       link tags in the head section.

    3. The consumer site sends the browser a redirect to the identity
       server.  This is the authentication request as described in the
       OpenID specification.

    4. The identity server's site sends the browser a redirect back to
       the consumer site.  This redirect contains the server's
       response to the authentication request.

    The most important part of the flow to note is the consumer's site
    must handle two separate HTTP requests in order to perform the
    full identity check.

LIBRARY DESIGN
==============

    This consumer library is designed with that flow in mind.  The
    goal is to make it as easy as possible to perform the above steps
    securely.

    At a high level, there are two important parts in the consumer
    library.  The first important part is this module, which contains
    the interface to actually use this library.  The second is the
    C{openid.consumer.stores} module, which describes the interface to
    use if you need to create a custom method for storing the state
    this library needs to maintain between requests.

    In general, the scond part is less important for users of the
    library to know about, as several implementations are provided
    which cover a wide variety of situations in which consumers may
    use the library.

    This module contains a class, C{OpenIDConsumer}, with methods
    corresponding to the actions necessary in each of steps 2, 3, and
    4 listed in the overview.  Use of this library should be as easy
    as creating an C{OpenIDConsumer} instance and calling the methods
    appropriate for the action the site wants to take.

STORES AND DUMB MODE
====================

    OpenID is a protocol that works best when the consumer site is
    able to store some state.  This is the normal mode of operation
    for the protocol, and is sometimes referred to as smart mode.
    There is also a fallback mode, known as dumb mode, which is
    available when the consumer site is not able to store state.  This
    mode should be avoided when possible, as it leaves the
    implementation more vulnerable to replay attacks.

    The mode the library works in for normal operation is determined
    by the store that it is given.  The store is an abstraction that
    handles the data that the consumer needs to manage between http
    requests in order to operate efficiently and securely.

    Several store implementation are provided, and the interface is
    fully documented so that custom stores can be used as well.  See
    the documentation for the C{OpenIDConsumer} class for more
    information on the interface for stores.  The concrete
    implementations that are provided allow the consumer site to store
    the necessary data in several different ways: in the filesystem,
    in a MySQL database, or in an SQLite database.

    There is an additional concrete store provided that puts the
    system in dumb mode.  This is not recommended, as it removes the
    library's ability to stop replay attacks reliably.  It still uses
    time-based checking to make replay attacks only possible within a
    small window, but they remain possible within that window.  This
    store should only be used if the consumer site has no way to store
    data between requests at all.

IMMEDIATE MODE
==============

    In the flow described above, there's a step which may occur if the
    user needs to confirm to the identity server that it's ok to
    authorize his or her identity.  The server may draw pages asking
    for information from the user before it redirects the browser back
    to the consumer's site.  This is generally transparent to the
    consumer site, so it is typically ignored as an implementation
    detail.

    There can be times, however, where the consumer site wants to get
    a response immediately.  When this is the case, the consumer can
    put the library in immediate mode.  In immediate mode, there is an
    extra response possible from the server, which is essentially the
    server reporting that it doesn't have enough information to answer
    the question yet.  In addition to saying that, the identity server
    provides a URL to which the user can be sent to provide the needed
    information and let the server finish handling the original
    request.

USING THIS LIBRARY
==================

    Use of this library is a straightforward process.
"""

__all__ = ['SUCCESS', 'FAILURE', 'SETUP_NEEDED', 'HTTP_FAILURE', 'PARSE_ERROR',
           'OpenIDAuthRequest', 'OpenIDConsumer']

from openid.consumer.impl import \
     SUCCESS, FAILURE, SETUP_NEEDED, PARSE_ERROR, HTTP_FAILURE, \
     OpenIDAuthRequest

class OpenIDConsumer(object):
    def __init__(self, store, fetcher=None, immediate=False):
        if fetcher is None:
            from openid.consumer.fetchers import getHTTPFetcher
            fetcher = getHTTPFetcher()

        from openid.consumer.impl import OpenIDConsumerImpl
        self.impl = OpenIDConsumerImpl(store, immediate, fetcher)

    def beginAuth(self, user_url):
        return self.impl.beginAuth(user_url)

    def constructRedirect(self, auth_request, return_to, trust_root):
        return self.impl.constructRedirect(auth_request, return_to, trust_root)

    def completeAuth(self, token, query):
        return self.impl.processServerResponse(token, query)

# XXX: left this in for convenience of copying comments

## class OpenIDConsumer(object):
##     """
##     """

##     def __init__(self, store, immediate=False, fetcher=None):
##         """
##         This method initializes a new OpenIDConsumer instance.  Users
##         of this OpenID consumer library need to create on
##         OpenIDConsumer instance to act as their gateway to the
##         library.

##         OpenIDConsumer instances store no per-request state, so they
##         may be used repeatedly in cases where they can remain in
##         memory between requests to the server.

##         @param store: This must be an object that implements the
##             interface in C{openid.consumer.stores.OpenIDStore}.
##             Several concrete implementations are provided, to cover
##             most common use cases.  For stores backed by MySQL or
##             SQLite, see the openid.consumer.sqlstore package.  For a
##             filesystem-backed store, see the
##             C{openid.consumer.filestore} package.

##             As a last resort, if it isn't possible for the server to
##             store state at all, an instance of
##             C{openid.consumer.stores.DumbStore} can be used.  This
##             should be an absolute last resort, though, as it makes the
##             consumer vulnerable to replay attacks over the lifespan of
##             the tokens the library creates.  See XXX for more
##             information on controlling the lifespan of tokens.

##         @type store: C{openid.consumer.stores.OpenIDStore}

##         @param immediate: This is an optional boolean value.  It
##             controls whether the library uses immediate mode, as
##             explained in the module description.  The default value is
##             False, which disables immediate mode.

##         @type immediate: C{bool}

##         @param fetcher: This is an optional instance of
##             C{openid.consumer.fetchers.OpenIDHTTPFetcher}.  If
##             present, the provided fetcher is used by the library to
##             fetch user's identity pages and make direct requests to
##             the identity server.  If it's not present, a default
##             fetcher is used.  The default fetcher uses curl if the
##             pycurl bindings are available, and uses urllib if not.

##         @type fetcher: C{openid.consumer.fetchers.OpenIDHTTPFetcher}

##         @return: As an initializer, this method has no return value.
##         """
##         if fetcher is None:
##             from openid.consumer.fetchers import getHTTPFetcher
##             fetcher = getHTTPFetcher()

##         from openid.consumer.impl import OpenIDConsumerImpl
##         self.impl = OpenIDConsumerImpl(store, immediate, fetcher)

##     def constructRedirect(self, proxy):
##         """
##         This method is called by the user of the consumer library to
##         construct the redirect URL used in step 2 of the flow
##         described above.  The user who is authenticating themselves
##         via OpenID should be sent a redirect to the generated URL.

##         First, the user's claimed identity page is fetched, to
##         determine their identity server.  Second, unless the library
##         is using a dumb store, it checks to see if it has an
##         association with that identity server, and creates and stores
##         one if it does not.

##         It then generates a signed token for this authentication
##         transaction, which contains a timestamp, a nonce, and the
##         information needed to finish the transaction.  This token
##         is passed in to the C{XXX} method, .

##         Finally, if all those steps completed successfully, the
##         generated URL is returned.  Otherwise C{None} is returned.

##         Calling this method can result in the following other methods
##         being called.
##         * XXX list of methods called indirectly

##         @param proxy: This is an object implementing the
##             C{OpenIDProxy} interface which can be used to help with
##             app-specific parts of constructing the redirect URL.

##         @type proxy: This is an instance of an object implementing the
##             C{OpenIDProxy} interface.

##         @return: This method returns a string containing the URL to
##             redirect to when such a URL is successfully constructed.

##             It returns C{None} when no such URL can be constructed.

##         @raise Exception: This method does not handle any exceptions
##             raised by the fetcher, the store, or any of the proxy's
##             methods that it calls.

##             It raises no exceptions itself.
##         """
##         return self.impl.constructRedirect(proxy)

##     def processServerResponse(self, proxy):
##         """
##         returns the value returned by whichever of the proxy's
##         handler methods was invoked

##         @param proxy: This is an object implementing the
##             C{OpenIDProxy} interface which can be used to help with
##             app-specific parts of constructing the redirect URL.

##         @type proxy: This is an instance of an object implementing the
##             C{OpenIDProxy} interface.

##         @return: During the course of the execution of this method, it
##             calls exactly one of the callback methods in the C{XXX}
##             object passed in to it.  The value returned by that
##             callback method is returned by this method.

##         @raise Exception: This method does not handle any exceptions
##             raised by the fetcher, the store, or any of the proxy's
##             methods that it calls.

##             It raises no exceptions itself.
##         """
##         return self.impl.processServerResponse(proxy)

## class OpenIDStartAuth(object):
##     """This object proxies between the library and an application for
##     initiating an OpenID conversation."""

##     def getUserInput(self):
##         """
##         This method is called by the OpenID consumer library

##         @return: This method returns the value the user gave as their
##             identity url.  If there is no such value available, it
##             returns C{None}.
##         """
##         raise NotImplementedError

##     def getTrustRoot(self):
##         """This method returns the server's trust root, or if the
##         server doesn't wish to specify a trust root, it returns None.
##         This is called during constructRedirect."""
##         raise NotImplementedError

##     def beginRedirect(self, token):
##         """Save the state of this request so that it can be restored
##         for completing the OpenID authentication. This method is
##         called once the library has contacted an identity server and
##         is ready to send the user there to confirm his authentication.

##         @param token: A string that allows the library to resume the
##             request and verify its validity. You must be able to
##             restore this token in the next step of authentication. The
##             token may be passed through the returned URL or may be
##             saved in the user's session.
##         @type token: str

##         @return: A URL that you can use to restore the state of this
##             request to complete the OpenID authentication. This URL
##             must be under the trust root that is returned by
##             getTrustRoot.
##         @rtype: str
##         """
##         raise NotImplementedError

##     def log(self, message):
##         pass

## class OpenIDFinishAuth(object):
##     """This is a proxy the openid library will use to get information
##     from its environment."""

##     def finishRedirect(self):
##         """Restore the state of the application so that it can
##         complete the OpenID request. This method is the first method
##         called when processing this request.

##         @return: The token that was supplied to beginRedirect when
##             this conversation was started. Return None if you are
##             unable to restore the token.

##         @rtype: str or NoneType
##         """
##         raise NotImplementedError

##     def getParameters(self):
##         """This method returns all the query parameters that the
##         server sent back to the consumer.  It returns them as a
##         dictionary from full parameter name (ie, including the
##         'openid.' prefix) to the parameter value, both unescaped from
##         whatever urlencoding was applied to them.  Used in
##         processServerResponse."""
##         raise NotImplementedError

##     def verifyReturnTo(self, return_to):
##         """Used by processServerResponse.  Returns true if the
##         return_to url matches the url this request came to, including
##         query args.  Otherwise returns false."""
##         raise NotImplementedError

##     def log(self, message):
##         pass

##     # The following methods are all callbacks used by
##     # processServerResponse and checkAuth.  For all of them, their
##     # return values are passed through, and returned from the call to
##     # OpenIDConsumer that led to them being called.  During normal
##     # execution (no exception raised) exactly one of these will be
##     # called per call to processServerResponse or checkAuth.

##     def loginGood(self, normalized_id):
##         """Called when a login is successful.  normalized_id is the
##         openid that was authenticated, after normalizing and following
##         redirects.  Any value returned by this method is returned by
##         the call to processServerRequest or checkAuth that led to this
##         method being called."""
##         raise NotImplementedError

##     def loginCancelled(self):
##         """Called when a user cancels a login.  No additional
##         information is included, so none is passed in.  Any value
##         returned by this method is returned by the call to
##         processServerRequest that led to this method being called."""
##         raise NotImplementedError

##     def loginFailure(self, normalized_id):
##         """Called when a login fails.  This usually indicates either a
##         malfunctioning server, an attempt to forge a login, or an
##         association with a server getting lost or expiring during the
##         transaction.  As it's generally impossible to determine which,
##         no additional information can be provided.  Any value returned
##         by this method is returned by the call to processServerRequest
##         or checkAuth that led to this method being called."""
##         raise NotImplementedError

##     def setupNeeded(self, user_setup_url):
##         """Called when the consumer is in immediate mode and the
##         server returns a user_setup_url.  Any value returned by this
##         method is returned by the call to processServerRequest that
##         led to this method being called."""
##         raise NotImplementedError
<commit_msg>[project @ doc project continuing]<commit_after>"""
This module is intended to document the main interface with the OpenID
consumer libary.  The only part of the library which has to be used
and isn't documented in full here is the store required to create an
C{OpenIDConsumer instance}.  More on the abstract store type and
concrete implementations of it that are provided in the documentation
for the C{__init__} method of the C{OpenIDConsumer} class.

OVERVIEW
========

    The OpenID identity verification process most commonly uses the
    following steps, as visible to the user of this library:

    1. The user enters their OpenID into a field on the consumer's
       site, and hits a log in button.

    2. The consumer site checks that the entered URL describes an
       OpenID page by fetching it and looking for appropriate
       link tags in the head section.

    3. The consumer site sends the browser a redirect to the identity
       server.  This is the authentication request as described in the
       OpenID specification.

    4. The identity server's site sends the browser a redirect back to
       the consumer site.  This redirect contains the server's
       response to the authentication request.

    The most important part of the flow to note is the consumer's site
    must handle two separate HTTP requests in order to perform the
    full identity check.

LIBRARY DESIGN
==============

    This consumer library is designed with that flow in mind.  The
    goal is to make it as easy as possible to perform the above steps
    securely.

    At a high level, there are two important parts in the consumer
    library.  The first important part is this module, which contains
    the interface to actually use this library.  The second is the
    C{openid.consumer.stores} module, which describes the interface to
    use if you need to create a custom method for storing the state
    this library needs to maintain between requests.

    In general, the scond part is less important for users of the
    library to know about, as several implementations are provided
    which cover a wide variety of situations in which consumers may
    use the library.

    This module contains a class, C{OpenIDConsumer}, with methods
    corresponding to the actions necessary in each of steps 2, 3, and
    4 listed in the overview.  Use of this library should be as easy
    as creating an C{OpenIDConsumer} instance and calling the methods
    appropriate for the action the site wants to take.

STORES AND DUMB MODE
====================

    OpenID is a protocol that works best when the consumer site is
    able to store some state.  This is the normal mode of operation
    for the protocol, and is sometimes referred to as smart mode.
    There is also a fallback mode, known as dumb mode, which is
    available when the consumer site is not able to store state.  This
    mode should be avoided when possible, as it leaves the
    implementation more vulnerable to replay attacks.

    The mode the library works in for normal operation is determined
    by the store that it is given.  The store is an abstraction that
    handles the data that the consumer needs to manage between http
    requests in order to operate efficiently and securely.

    Several store implementation are provided, and the interface is
    fully documented so that custom stores can be used as well.  See
    the documentation for the C{OpenIDConsumer} class for more
    information on the interface for stores.  The concrete
    implementations that are provided allow the consumer site to store
    the necessary data in several different ways: in the filesystem,
    in a MySQL database, or in an SQLite database.

    There is an additional concrete store provided that puts the
    system in dumb mode.  This is not recommended, as it removes the
    library's ability to stop replay attacks reliably.  It still uses
    time-based checking to make replay attacks only possible within a
    small window, but they remain possible within that window.  This
    store should only be used if the consumer site has no way to store
    data between requests at all.

IMMEDIATE MODE
==============

    In the flow described above, there's a step which may occur if the
    user needs to confirm to the identity server that it's ok to
    authorize his or her identity.  The server may draw pages asking
    for information from the user before it redirects the browser back
    to the consumer's site.  This is generally transparent to the
    consumer site, so it is typically ignored as an implementation
    detail.

    There can be times, however, where the consumer site wants to get
    a response immediately.  When this is the case, the consumer can
    put the library in immediate mode.  In immediate mode, there is an
    extra response possible from the server, which is essentially the
    server reporting that it doesn't have enough information to answer
    the question yet.  In addition to saying that, the identity server
    provides a URL to which the user can be sent to provide the needed
    information and let the server finish handling the original
    request.

USING THIS LIBRARY
==================

    Use of this library is a straightforward process.
"""

__all__ = ['SUCCESS', 'FAILURE', 'SETUP_NEEDED', 'HTTP_FAILURE', 'PARSE_ERROR',
           'OpenIDAuthRequest', 'OpenIDConsumer']

from openid.consumer.impl import \
     SUCCESS, FAILURE, SETUP_NEEDED, PARSE_ERROR, HTTP_FAILURE, \
     OpenIDAuthRequest

class OpenIDConsumer(object):
    """
    @ivar impl: balh
    """

    def __init__(self, store, fetcher=None, immediate=False):
        """
        This method initializes a new C{OpenIDConsumer} instance.
        Users of the OpenID consumer library need to create an
        C{OpenIDConsumer} instance to access the library.

        OpenIDConsumer instances store no per-request state, so they
        may be used repeatedly when it is desired.

        @param store: This must be an object that implements the
            interface in C{openid.consumer.stores.OpenIDStore}.
            Several concrete implementations are provided, to cover
            most common use cases.  For stores backed by MySQL or
            SQLite, see the openid.consumer.sqlstore package.  For a
            filesystem-backed store, see the
            C{openid.consumer.filestore} package.

            As a last resort, if it isn't possible for the server to
            store state at all, an instance of
            C{openid.consumer.stores.DumbStore} can be used.  This
            should be an absolute last resort, though, as it makes the
            consumer vulnerable to replay attacks over the lifespan of
            the tokens the library creates.  See L{impl} for
            information on controlling the lifespan of those tokens.

        @type store: C{openid.consumer.stores.OpenIDStore}

        @param fetcher: This is an optional instance of
            C{openid.consumer.fetchers.OpenIDHTTPFetcher}.  If
            present, the provided fetcher is used by the library to
            fetch user's identity pages and make direct requests to
            the identity server.  If it's not present, a default
            fetcher is used.  The default fetcher uses curl if the
            pycurl bindings are available, and uses urllib if not.

        @type fetcher: C{openid.consumer.fetchers.OpenIDHTTPFetcher}

        @param immediate: This is an optional boolean value.  It
            controls whether the library uses immediate mode, as
            explained in the module description.  The default value is
            False, which disables immediate mode.

        @type immediate: C{bool}
        """
        if fetcher is None:
            from openid.consumer.fetchers import getHTTPFetcher
            fetcher = getHTTPFetcher()

        from openid.consumer.impl import OpenIDConsumerImpl
        self.impl = OpenIDConsumerImpl(store, immediate, fetcher)

    def beginAuth(self, user_url):
        """
        First, the user's claimed identity page is fetched, to
        determine their identity server.  Second, unless the library
        is using a dumb store, it checks to see if it has an
        association with that identity server, and creates and stores
        one if it does not.

        It then generates a signed token for this authentication
        transaction, which contains a timestamp, a nonce, and the
        information needed to finish the transaction.  This token
        is passed in to the C{XXX} method, .

        """
        return self.impl.beginAuth(user_url)

    def constructRedirect(self, auth_request, return_to, trust_root):
        """
        This method is called by the user of the consumer library to
        construct the redirect URL used in step 3 of the flow
        described in the overview.  The generated redirect should be
        sent to the browser which initiated the authorization request.

        @param auth_request: This must be an C{OpenIDAuthRequest}
            instance which was returned from a previous call to
            C{L{beginAuth}}.  It contains information found during the
            beginAuth call which is needed to build the redirect URL.

        @type auth_request: C{OpenIDAuthRequest}

        @param return_to: This is the URL that will be included in the
            generated redirect as the URL the OpenID server will send
            its response to.  The URL passed in must handle OpenID
            authentication responses.

        @type return_to: C{str}

        @param trust_root: This is a URL that will be sent to the
            server to identify this site.  U{The OpenID
            spec<http://www.openid.net/specs.bml#mode-checkid_immediate>}
            has more information on what the trust_root value is for
            and what its form can be.

        @type trust_root: C{str}

        @return: This method returns a string containing the URL to
            redirect to when such a URL is successfully constructed.

            It returns C{None} when no such URL can be constructed.

        @raise Exception: This method does not handle any exceptions
            raised by the store it is using.

            It raises no exceptions itself.
        """
        return self.impl.constructRedirect(auth_request, return_to, trust_root)

    def completeAuth(self, token, query):
        return self.impl.processServerResponse(token, query)

# XXX: left this in for convenience of copying comments

## class OpenIDConsumer(object):
##     """
##     """

##     def __init__(self, store, immediate=False, fetcher=None):
##         if fetcher is None:
##             from openid.consumer.fetchers import getHTTPFetcher
##             fetcher = getHTTPFetcher()

##         from openid.consumer.impl import OpenIDConsumerImpl
##         self.impl = OpenIDConsumerImpl(store, immediate, fetcher)

##     def constructRedirect(self, proxy):
##         return self.impl.constructRedirect(proxy)

##     def processServerResponse(self, proxy):
##         """
##         returns the value returned by whichever of the proxy's
##         handler methods was invoked

##         @param proxy: This is an object implementing the
##             C{OpenIDProxy} interface which can be used to help with
##             app-specific parts of constructing the redirect URL.

##         @type proxy: This is an instance of an object implementing the
##             C{OpenIDProxy} interface.

##         @return: During the course of the execution of this method, it
##             calls exactly one of the callback methods in the C{XXX}
##             object passed in to it.  The value returned by that
##             callback method is returned by this method.

##         @raise Exception: This method does not handle any exceptions
##             raised by the fetcher, the store, or any of the proxy's
##             methods that it calls.

##             It raises no exceptions itself.
##         """
##         return self.impl.processServerResponse(proxy)

## class OpenIDStartAuth(object):
##     """This object proxies between the library and an application for
##     initiating an OpenID conversation."""

##     def getUserInput(self):
##         """
##         This method is called by the OpenID consumer library

##         @return: This method returns the value the user gave as their
##             identity url.  If there is no such value available, it
##             returns C{None}.
##         """
##         raise NotImplementedError

##     def getTrustRoot(self):
##         """This method returns the server's trust root, or if the
##         server doesn't wish to specify a trust root, it returns None.
##         This is called during constructRedirect."""
##         raise NotImplementedError

##     def beginRedirect(self, token):
##         """Save the state of this request so that it can be restored
##         for completing the OpenID authentication. This method is
##         called once the library has contacted an identity server and
##         is ready to send the user there to confirm his authentication.

##         @param token: A string that allows the library to resume the
##             request and verify its validity. You must be able to
##             restore this token in the next step of authentication. The
##             token may be passed through the returned URL or may be
##             saved in the user's session.
##         @type token: str

##         @return: A URL that you can use to restore the state of this
##             request to complete the OpenID authentication. This URL
##             must be under the trust root that is returned by
##             getTrustRoot.
##         @rtype: str
##         """
##         raise NotImplementedError

##     def log(self, message):
##         pass

## class OpenIDFinishAuth(object):
##     """This is a proxy the openid library will use to get information
##     from its environment."""

##     def finishRedirect(self):
##         """Restore the state of the application so that it can
##         complete the OpenID request. This method is the first method
##         called when processing this request.

##         @return: The token that was supplied to beginRedirect when
##             this conversation was started. Return None if you are
##             unable to restore the token.

##         @rtype: str or NoneType
##         """
##         raise NotImplementedError

##     def getParameters(self):
##         """This method returns all the query parameters that the
##         server sent back to the consumer.  It returns them as a
##         dictionary from full parameter name (ie, including the
##         'openid.' prefix) to the parameter value, both unescaped from
##         whatever urlencoding was applied to them.  Used in
##         processServerResponse."""
##         raise NotImplementedError

##     def verifyReturnTo(self, return_to):
##         """Used by processServerResponse.  Returns true if the
##         return_to url matches the url this request came to, including
##         query args.  Otherwise returns false."""
##         raise NotImplementedError

##     def log(self, message):
##         pass

##     # The following methods are all callbacks used by
##     # processServerResponse and checkAuth.  For all of them, their
##     # return values are passed through, and returned from the call to
##     # OpenIDConsumer that led to them being called.  During normal
##     # execution (no exception raised) exactly one of these will be
##     # called per call to processServerResponse or checkAuth.

##     def loginGood(self, normalized_id):
##         """Called when a login is successful.  normalized_id is the
##         openid that was authenticated, after normalizing and following
##         redirects.  Any value returned by this method is returned by
##         the call to processServerRequest or checkAuth that led to this
##         method being called."""
##         raise NotImplementedError

##     def loginCancelled(self):
##         """Called when a user cancels a login.  No additional
##         information is included, so none is passed in.  Any value
##         returned by this method is returned by the call to
##         processServerRequest that led to this method being called."""
##         raise NotImplementedError

##     def loginFailure(self, normalized_id):
##         """Called when a login fails.  This usually indicates either a
##         malfunctioning server, an attempt to forge a login, or an
##         association with a server getting lost or expiring during the
##         transaction.  As it's generally impossible to determine which,
##         no additional information can be provided.  Any value returned
##         by this method is returned by the call to processServerRequest
##         or checkAuth that led to this method being called."""
##         raise NotImplementedError

##     def setupNeeded(self, user_setup_url):
##         """Called when the consumer is in immediate mode and the
##         server returns a user_setup_url.  Any value returned by this
##         method is returned by the call to processServerRequest that
##         led to this method being called."""
##         raise NotImplementedError
<|endoftext|>


<commit_before>#!/usr/bin/env python
# encoding: utf-8
__author__ = 'Kxrr'

import re

with open('0004.txt', 'r') as f:
    dictResult = {}

    # Find the letters each line
    for line in f.readlines():
        listMatch = re.findall('[a-zA-Z]+', line.lower()) # remember to lower the letters

    # Count
        for eachLetter in listMatch:
            eachLetterCount = len(re.findall(eachLetter, line.lower()))
            dictResult[eachLetter] = dictResult.get(eachLetter, 0) + eachLetterCount

    # Sort the result
    result = sorted(dictResult.items(), key=lambda d: d[1], reverse=True)
    for each in result:
        print each

"""
('a', 239)
('he', 172)
('s', 121)
('i', 97)
('the', 93)
('to', 86)
('in', 70)
('t', 44)
('his', 41)
('it', 31)
('and', 29)
('o', 29)
('at', 21)
('of', 20)
('saito', 19)
('on', 19)
('that', 17)
('is', 16)
('for', 16)
('was', 16)
('him', 14)
('so', 12)
('image', 12)
('parents', 12)
('as', 12)
('william', 11)
('japanese', 10)
('which', 10)
('d', 10)
('says', 8)
('japan', 8)
('mr', 8)
('but', 8)
('an', 8)
('me', 7)
('caption', 7)
('are', 7)
('doctor', 7)
('has', 7)
('be', 7)
('teacher', 6)
('believes', 6)
('their', 6)
('copyright', 6)
('just', 6)
('us', 6)
('you', 6)
('this', 5)
('give', 5)
('years', 5)
('start', 5)
('about', 5)
('age', 5)
('changed', 4)
('would', 4)
('more', 4)
('entrepreneurs', 4)
('media', 4)
('help', 4)
('they', 4)
('out', 4)
('given', 4)
('m', 4)
('country', 4)
('firm', 4)
('while', 4)
('technology', 4)
('make', 4)
('opportunity', 4)
('old', 4)
('people', 4)
('up', 4)
('advantage', 3)
('entrepreneurial', 3)
('experience', 3)
('makes', 3)
('from', 3)
('my', 3)
('something', 3)
('such', 3)
('still', 3)
('personal', 3)
('have', 3)
('also', 3)
('take', 3)
('who', 3)
('computer', 3)
('life', 3)
('player', 3)
('things', 3)
('had', 3)
('become', 3)
('because', 3)
('back', 3)
('home', 3)
('by', 3)
('software', 3)
('when', 3)
('founded', 2)
('young', 2)
('teaching', 2)
('then', 2)
('focused', 2)
('two', 2)
('successful', 2)
('company', 2)
('work', 2)
('can', 2)
('high', 2)
('how', 2)
('after', 2)
('main', 2)
('now', 2)
('always', 2)
('year', 2)
('special', 2)
('re', 2)
('wanted', 2)
('first', 2)
('one', 2)
('determined', 2)
('venture', 2)
('medical', 2)
('price', 2)
('paid', 2)
('microsoft', 2)
('particularly', 2)
('reveal', 2)
('view', 2)
('fail', 2)
('received', 2)
('became', 2)
('point', 2)
('speak', 2)
('different', 2)
('same', 2)
('well', 2)
('money', 2)
('intecur', 2)
('success', 2)
('security', 2)
('some', 2)
('born', 2)
('business', 2)
('own', 2)
('support', 2)
('with', 2)
('other', 2)
('time', 2)
('eldest', 1)
('issues', 1)
('specialising', 1)
('permanently', 1)
('worth', 1)
('activities', 1)
('retire', 1)
('very', 1)
('focus', 1)
('lynch', 1)
('bringing', 1)
('school', 1)
('level', 1)
('did', 1)
('companies', 1)
('relentlessly', 1)
('travels', 1)
('jump', 1)
('investment', 1)
('what', 1)
('giving', 1)
('section', 1)
('attitudes', 1)
('capital', 1)
('new', 1)
('firms', 1)
('unsurprisingly', 1)
('reporter', 1)
('never', 1)
('here', 1)
('change', 1)
('settling', 1)
('root', 1)
('suitable', 1)
('family', 1)
('aimed', 1)
('next', 1)
('taken', 1)
('overlooked', 1)
('today', 1)
('formally', 1)
('iris', 1)
('nguyen', 1)
('particular', 1)
('women', 1)
('word', 1)
('tokyo', 1)
('making', 1)
('heart', 1)
('india', 1)
('tab', 1)
('serial', 1)
('information', 1)
('end', 1)
('thing', 1)
('complaining', 1)
('hot', 1)
('unleash', 1)
('instead', 1)
('plans', 1)
('huge', 1)
('advice', 1)
('date', 1)
('grow', 1)
('adviser', 1)
('dream', 1)
('enter', 1)
('september', 1)
('over', 1)
('mission', 1)
('through', 1)
('motorcycle', 1)
('before', 1)
('continued', 1)
('actually', 1)
('stifling', 1)
('eventually', 1)
('good', 1)
('return', 1)
('emphasised', 1)
('emigrated', 1)
('fingerprint', 1)
('not', 1)
('bank', 1)
('university', 1)
('didn', 1)
('went', 1)
('mean', 1)
('doing', 1)
('hard', 1)
('pursuits', 1)
('society', 1)
('expect', 1)
('begun', 1)
('organised', 1)
('beyond', 1)
('network', 1)
('since', 1)
('suggesting', 1)
('weaknesses', 1)
('housed', 1)
('story', 1)
('admits', 1)
('reason', 1)
('california', 1)
('maths', 1)
('obligation', 1)
('training', 1)
('couldn', 1)
('could', 1)
('computers', 1)
('american', 1)
('incorporated', 1)
('think', 1)
('merrill', 1)
('carry', 1)
('friend', 1)
('selling', 1)
('hotel', 1)
('took', 1)
('than', 1)
('aims', 1)
('precocious', 1)
('risks', 1)
('taxis', 1)
('ran', 1)
('generation', 1)
('turned', 1)
('advising', 1)
('confesses', 1)
('able', 1)
('potential', 1)
('strengths', 1)
('most', 1)
('clear', 1)
('later', 1)
('sale', 1)
('investing', 1)
('definitely', 1)
('businessman', 1)
('typically', 1)
('fact', 1)
('saying', 1)
('queen', 1)
('biometrics', 1)
('lessons', 1)
('giant', 1)
('unfinished', 1)
('failed', 1)
('bases', 1)
('judged', 1)
('do', 1)
('soon', 1)
('feels', 1)
('cannot', 1)
('despite', 1)
('happiest', 1)
('textbooks', 1)
('jokes', 1)
('bad', 1)
('culture', 1)
('see', 1)
('result', 1)
('enhanced', 1)
('said', 1)
('lots', 1)
('various', 1)
('finger', 1)
('thrive', 1)
('terms', 1)
('importance', 1)
('sold', 1)
('boss', 1)
('suggested', 1)
('reverse', 1)
('loan', 1)
('writing', 1)
('revolutionising', 1)
('quit', 1)
('bbc', 1)
('three', 1)
('been', 1)
('judging', 1)
('expected', 1)
('both', 1)
('finished', 1)
('wake', 1)
('child', 1)
('worked', 1)
('spirit', 1)
('hoang', 1)
('these', 1)
('harder', 1)
('many', 1)
('fun', 1)
('played', 1)
('someone', 1)
('optimistic', 1)
('owes', 1)
('if', 1)
('complex', 1)
('largest', 1)
('several', 1)
('difficult', 1)
('driven', 1)
('moment', 1)
('changing', 1)
('english', 1)
('position', 1)
('left', 1)
('competitive', 1)
('schools', 1)
('grades', 1)
('immediately', 1)
('keen', 1)
('drive', 1)
('interested', 1)
('real', 1)
('government', 1)
('know', 1)
('press', 1)
('using', 1)
('pies', 1)
('advanced', 1)
('continue', 1)
('works', 1)
('fulfil', 1)
('often', 1)
('deal', 1)
('exposed', 1)
('cybersecurity', 1)
('hair', 1)
('recognition', 1)
('denies', 1)
('competitions', 1)
('leader', 1)
('run', 1)
('refused', 1)
('credits', 1)
('volunteering', 1)
('getting', 1)
('angeles', 1)
('ensure', 1)
('or', 1)
('raised', 1)
('previously', 1)
('into', 1)
('son', 1)
('entrepreneur', 1)
('consultancy', 1)
('female', 1)
('lesson', 1)
('your', 1)
('aren', 1)
('qualifying', 1)
('long', 1)
('los', 1)
('accountant', 1)
('way', 1)
('heritage', 1)
('buy', 1)
('becoming', 1)
('hoped', 1)
('failure', 1)
('gain', 1)
('hopes', 1)
('technologies', 1)
('junior', 1)
('raising', 1)
('made', 1)
('called', 1)
('proud', 1)
('moved', 1)
('direction', 1)
('globally', 1)
('seniority', 1)
('programmes', 1)
('mathematics', 1)
('really', 1)
('favourite', 1)
('felt', 1)
('helping', 1)
('invested', 1)
('cheque', 1)
('building', 1)
('lead', 1)
('calculates', 1)
('vietnam', 1)
('far', 1)
('starting', 1)
('once', 1)
"""<commit_msg>0004 solved<commit_after>#!/usr/bin/env python
# encoding: utf-8
__author__ = 'Kxrr'

import re

with open('0004.txt', 'r') as f:
    dictResult = {}

    # Find the letters each line
    for line in f.readlines():
        listMatch = re.findall('[a-zA-Z]+', line.lower()) # remember to lower the letters

    # Count
        for eachLetter in listMatch:
            eachLetterCount = len(re.findall(eachLetter, line.lower()))
            dictResult[eachLetter] = dictResult.get(eachLetter, 0) + eachLetterCount

    # Sort the result
    result = sorted(dictResult.items(), key=lambda d: d[1], reverse=True)
    for each in result:
        print each<|endoftext|>


<commit_before># -*- coding: utf8 -*-
# This file is part of PYBOSSA.
#
# Copyright (C) 2017 Scifabric LTD.
#
# PYBOSSA is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# PYBOSSA is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU Affero General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with PYBOSSA.  If not, see <http://www.gnu.org/licenses/>.

import json
from pybossa.jobs import create_onesignal_app, push_notification
from default import Test, with_context
from mock import patch, MagicMock, call
from factories import ProjectFactory, UserFactory, CategoryFactory
from pybossa.core import project_repo
from flask import url_for

class TestOnesignal(Test):

    def setUp(self):
        super(TestOnesignal, self).setUp()
        CategoryFactory.create()

    @with_context
    @patch('pybossa.jobs.PybossaOneSignal')
    def test_create_onesignal_no_config(self, mock_onesignal):
        """Test create_onesignal with no config works."""

        project = ProjectFactory.create()

        client = MagicMock()
        client.create_app.return_value = (200, 'OK',
                                          {'basic_auth': 'auth', 'id': 1})
        mock_onesignal.return_value = client

        create_onesignal_app(project.id)

        assert client.called is False

    @with_context
    @patch('pybossa.jobs.PybossaOneSignal')
    def test_push_notification_onesignal_no_config(self, mock_onesignal):
        """Test push_notification with no config works."""

        user = UserFactory.create()

        project = ProjectFactory.create(owner=user)

        blog = dict(title="hello", body="world", project_id=project.id)

        url = '/api/blogpost?api_key=%s' % user.api_key

        res = self.app.post(url, data=json.dumps(blog))

        assert res.status_code == 200, res.data
        assert mock_onesignal.called is False

    @with_context
    @patch('pybossa.jobs.PybossaOneSignal')
    @patch('pybossa.model.event_listeners.webpush_queue.enqueue')
    def test_create_onesignal(self, mock_queue, mock_onesignal):
        """Test create_onesignal with config works."""
        client = MagicMock()
        osdata = {'basic_auth': 'auth', 'id': 1}
        client.create_app.return_value = (200, 'OK', osdata)
        mock_onesignal.return_value = client

        with patch.dict(self.flask_app.config, {'ONESIGNAL_AUTH_KEY': 'key'}):
            user = UserFactory.create()
            project_data = dict(name='name', short_name='short_name',
                                description='desc',
                                long_description='long')
            url = '/api/project?api_key=%s' % user.api_key

            res = self.app.post(url, data=json.dumps(project_data))

            assert res.status_code == 200, res.data

            project = json.loads(res.data)

            mock_queue.assert_called_with(create_onesignal_app, project['id'])

            res = create_onesignal_app(project['id'])

            assert res[0] == 200, res
            assert res[1] == 'OK', res
            assert res[2]['id'] == 1, res

            new = project_repo.get(project['id'])

            assert new.info['onesignal'] == osdata, new.info
            assert new.info['onesignal_app_id'] == 1, new.info

    @with_context
    @patch('pybossa.jobs.PybossaOneSignal')
    @patch('pybossa.model.event_listeners.webpush_queue.enqueue')
    def test_push_notification(self, mock_queue, mock_onesignal):
        """Test push_notification with config works."""

        user = UserFactory.create()

        project = ProjectFactory.create(owner=user)

        blog = dict(title="hello", body="world", project_id=project.id)

        url = '/api/blogpost?api_key=%s' % user.api_key

        res = self.app.post(url, data=json.dumps(blog))

        blogdata = json.loads(res.data)

        assert res.status_code == 200, res.data

        contents = {"en": "New update!"}
        headings = {"en": blog['title']}
        launch_url = url_for('project.show_blogpost',
                             short_name=project.short_name,
                             id=blogdata['id'],
                             _external=True)
        web_buttons = [{"id": "read-more-button",
                        "text": "Read more",
                        "icon": "http://i.imgur.com/MIxJp1L.png",
                        "url": launch_url }]

        mock_queue.assert_called_with(push_notification,
                                      project_id=project.id,
                                      contents=contents,
                                      headings=headings,
                                      web_buttons=web_buttons,
                                      launch_url=launch_url)
<commit_msg>More tests.<commit_after># -*- coding: utf8 -*-
# This file is part of PYBOSSA.
#
# Copyright (C) 2017 Scifabric LTD.
#
# PYBOSSA is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# PYBOSSA is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU Affero General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with PYBOSSA.  If not, see <http://www.gnu.org/licenses/>.

import json
from pybossa.jobs import create_onesignal_app, push_notification
from default import Test, with_context
from mock import patch, MagicMock, call
from factories import ProjectFactory, UserFactory, CategoryFactory
from pybossa.core import project_repo
from flask import url_for

class TestOnesignal(Test):

    def setUp(self):
        super(TestOnesignal, self).setUp()
        CategoryFactory.create()

    @with_context
    @patch('pybossa.jobs.PybossaOneSignal')
    def test_create_onesignal_no_config(self, mock_onesignal):
        """Test create_onesignal with no config works."""

        project = ProjectFactory.create()

        client = MagicMock()
        client.create_app.return_value = (200, 'OK',
                                          {'basic_auth': 'auth', 'id': 1})
        mock_onesignal.return_value = client

        create_onesignal_app(project.id)

        assert client.called is False

    @with_context
    @patch('pybossa.jobs.PybossaOneSignal')
    def test_push_notification_onesignal_no_config(self, mock_onesignal):
        """Test push_notification with no config works."""

        user = UserFactory.create()

        project = ProjectFactory.create(owner=user)

        blog = dict(title="hello", body="world", project_id=project.id)

        url = '/api/blogpost?api_key=%s' % user.api_key

        res = self.app.post(url, data=json.dumps(blog))

        assert res.status_code == 200, res.data
        assert mock_onesignal.called is False

    @with_context
    @patch('pybossa.jobs.PybossaOneSignal')
    @patch('pybossa.model.event_listeners.webpush_queue.enqueue')
    def test_create_onesignal(self, mock_queue, mock_onesignal):
        """Test create_onesignal with config works."""
        client = MagicMock()
        osdata = {'basic_auth': 'auth', 'id': 1}
        client.create_app.return_value = (200, 'OK', osdata)
        mock_onesignal.return_value = client

        with patch.dict(self.flask_app.config, {'ONESIGNAL_AUTH_KEY': 'key'}):
            user = UserFactory.create()
            project_data = dict(name='name', short_name='short_name',
                                description='desc',
                                long_description='long')
            url = '/api/project?api_key=%s' % user.api_key

            res = self.app.post(url, data=json.dumps(project_data))

            assert res.status_code == 200, res.data

            project = json.loads(res.data)

            mock_queue.assert_called_with(create_onesignal_app, project['id'])

            res = create_onesignal_app(project['id'])

            assert res[0] == 200, res
            assert res[1] == 'OK', res
            assert res[2]['id'] == 1, res

            new = project_repo.get(project['id'])

            assert new.info['onesignal'] == osdata, new.info
            assert new.info['onesignal_app_id'] == 1, new.info

    @with_context
    @patch('pybossa.jobs.PybossaOneSignal')
    @patch('pybossa.model.event_listeners.webpush_queue.enqueue')
    def test_push_notification(self, mock_queue, mock_onesignal):
        """Test push_notification with config works."""

        user = UserFactory.create()

        project = ProjectFactory.create(owner=user)

        blog = dict(title="hello", body="world", project_id=project.id)

        url = '/api/blogpost?api_key=%s' % user.api_key

        res = self.app.post(url, data=json.dumps(blog))

        blogdata = json.loads(res.data)

        assert res.status_code == 200, res.data

        contents = {"en": "New update!"}
        headings = {"en": blog['title']}
        launch_url = url_for('project.show_blogpost',
                             short_name=project.short_name,
                             id=blogdata['id'],
                             _external=True)
        web_buttons = [{"id": "read-more-button",
                        "text": "Read more",
                        "icon": "http://i.imgur.com/MIxJp1L.png",
                        "url": launch_url }]

        mock_queue.assert_called_with(push_notification,
                                      project_id=project.id,
                                      contents=contents,
                                      headings=headings,
                                      web_buttons=web_buttons,
                                      launch_url=launch_url)

    @with_context
    @patch('pybossa.jobs.PybossaOneSignal.push_msg')
    def test_push_notification_method(self, mock_onesignal):
        """Test push_notification method alone."""
        project = ProjectFactory.create(info=dict(onesignal=dict(id=1, basic_auth_key='key')))
        mock_onesignal.return_value = "msg"

        contents = {"en": "New update!"}
        headings = {"en": 'title'}
        launch_url = 'https://hola.com'
        web_buttons = [{"id": "read-more-button",
                        "text": "Read more",
                        "icon": "http://i.imgur.com/MIxJp1L.png",
                        "url": launch_url }]

        res = push_notification(project.id,
                                contents=contents,
                                headings=headings,
                                web_buttons=web_buttons,
                                launch_url=launch_url)

        assert mock_onesignal.called
        assert res == "msg", res
<|endoftext|>


<commit_before>import argparse
import os

from enum import Enum

from htmresearch.encoders import EncoderTypes
from htmresearch.frameworks.nlp.classification_model import ClassificationModel
from htmresearch.frameworks.nlp.classify_fingerprint import (
  ClassificationModelFingerprint)
from htmresearch.frameworks.nlp.classify_htm import ClassificationModelHTM
from htmresearch.frameworks.nlp.classify_keywords import (
  ClassificationModelKeywords)
from htmresearch.frameworks.nlp.classify_windows import (
  ClassificationModelWindows)
from htmresearch.support.csv_helper import readCSV
from htmresearch.support.text_preprocess import TextPreprocess

class ModelTypes(Enum):
  CioWordFingerprint = ClassificationModelFingerprint
  CioDocumentFingerprint = ClassificationModelFingerprint
  CioWindows = ClassificationModelWindows
  Keywords = ClassificationModelKeywords
  HTMNetwork = ClassificationModelHTM

class ModelSimilarityMetrics(Enum):
  pctOverlapOfInput = "pctOverlapOfInput"

class ImbuError(Exception):
  pass

class ImbuUnableToLoadModelError(ImbuError):
  def __init__(self, exc):
    self.exc = exc

class Imbu(object):

  defaultSimilarityMetric = ModelSimilarityMetrics.pctOverlapOfInput
  defaultModelType = ModelTypes.HTMNetwork

  def __init__(self, cacheRoot, dataPath, loadPath, savePath,
      modelSimilarityMetric):
    self.cacheRoot = cacheRoot
    self.modelSimilarityMetric = (
      modelSimilarityMetric or self.defaultSimilarityMetric
    )
    self.loadPath = loadPath
    self.savePath = savePath
    self.dataPath = dataPath

  def __repr__(self):
    return ("Imbu<cacheRoot={cacheRoot}, dataPath={dataPath}, "
            "loadPath={loadPath}, savePath={savePath}, "
            "modelSimilarityMetric={modelSimilarityMetric}>"
            .format(**self.__dict__))

  def modelFactory(self, modelType, **kwargs):

    modelType = modelType or self.defaultModelType

    if modelType is ModelTypes.CioWordFingerprint:
      model = modelType.value(retina=kwargs.get("retina"),
                              apiKey=kwargs.get("apiKey"),
                              fingerprintType=EncoderTypes.word,
                              modelDir=self.savePath,
                              cacheRoot=self.cacheRoot,
                              classifierMetric=self.modelSimilarityMetric.value)

    elif modelType is ModelTypes.CioDocumentFingerprint:
      model = modelType.value(retina=kwargs.get("retina"),
                              apiKey=kwargs.get("apiKey"),
                              fingerprintType=EncoderTypes.document,
                              modelDir=self.savePath,
                              cacheRoot=self.cacheRoot,
                              classifierMetric=self.modelSimilarityMetric.value)

    elif modelType is ModelTypes.HTMNetwork:
      raise NotImplementedError("HTMNetwork model type is not implemented.")

    else:
      if modelType not in ModelTypes:
        raise NotImplementedError()

      model = modelType.value(modelDir=self.savePath,
                              classifierMetric=self.modelSimilarityMetric.value)

    model.verbosity = 0
    model.numLabels = 0

    return model

  def loadModel(self):
    try:
      return ClassificationModel.load(self.loadPath)
    except IOError as exc:
      raise ImbuUnableToLoadModelError(exc)

  def _loadTrainingData(self):
    return readCSV(self.dataPath, numLabels=0)

  def train(self, model):
    dataDict = self._loadTrainingData()
    raise NotImplementedError() # Not sure what to do here any more

def main():
  parser = argparse.ArgumentParser()

  parser.add_argument("--cacheRoot",
                      help="Root directory in which to cache encodings")
  parser.add_argument("--modelSimilarityMetric",
                      choices={metric.name
                               for metric in ModelSimilarityMetrics},
                      default=Imbu.defaultSimilarityMetric.name,
                      help="Classifier metric")
  parser.add_argument("--dataPath",
                      help="Path to data CSV; samples must be in column w/ "
                           "header 'Sample'; see readCSV() for more.",
                      required=True)
  parser.add_argument("--modelName",
                      choices={modelType.name for modelType in ModelTypes},
                      default=Imbu.defaultModelType.name,
                      type=str,
                      help="Name of model class. Also used for model results "
                           "directory and pickle checkpoint.")
  parser.add_argument("--loadPath",
                      default="",
                      type=str,
                      help="Path from which to load a serialized model.")
  parser.add_argument("--savePath",
                      default="",
                      type=str,
                      help="Path to save the serialized model.")
  args = parser.parse_args()

  imbu = Imbu(
    cacheRoot=args.cacheRoot,
    modelSimilarityMetric=getattr(ModelSimilarityMetrics,
                                  args.modelSimilarityMetric),
    dataPath=args.dataPath,
    loadPath=args.loadPath,
    savePath=args.savePath
  )

  if args.loadPath:
    model = imbu.loadModel()
  else:
    model = imbu.modelFactory(getattr(ModelTypes, args.modelName),
                              retina=os.environ["IMBU_RETINA_ID"],
                              apiKey=os.environ["CORTICAL_API_KEY"])

  imbu.train(model)

  if args.savePath:
    model.save()

  # Query the model.
  printTemplate = "{0:<10}|{1:<30}"
  while True:
    print "Now we query the model for samples (quit with 'q')..."

    query = raw_input("Enter a query: ")

    if query == "q":
      break

    sortedDistances = model.queryModel(query)

    print printTemplate.format("Sample ID", "Distance from query")

    for sID, dist in sortedDistances:
      print printTemplate.format(sID, dist)

if __name__ == "__main__":
  main()
<commit_msg>IMBU-70 Additional implementation details re: formal interface<commit_after>import argparse
import os

from enum import Enum

from htmresearch.encoders import EncoderTypes
from htmresearch.frameworks.nlp.classification_model import ClassificationModel
from htmresearch.frameworks.nlp.classify_fingerprint import (
  ClassificationModelFingerprint)
from htmresearch.frameworks.nlp.classify_htm import ClassificationModelHTM
from htmresearch.frameworks.nlp.classify_keywords import (
  ClassificationModelKeywords)
from htmresearch.frameworks.nlp.classify_windows import (
  ClassificationModelWindows)
from htmresearch.support.csv_helper import readCSV
from htmresearch.support.text_preprocess import TextPreprocess

class ImbuModelTypes(Enum):
  CioWordFingerprint = ClassificationModelFingerprint
  CioDocumentFingerprint = ClassificationModelFingerprint
  CioWindows = ClassificationModelWindows
  Keywords = ClassificationModelKeywords
  HTMNetwork = ClassificationModelHTM

class ModelSimilarityMetrics(Enum):
  pctOverlapOfInput = "pctOverlapOfInput"

class ImbuError(Exception):
  pass

class ImbuUnableToLoadModelError(ImbuError):
  def __init__(self, exc):
    self.exc = exc

class ImbuModels(object):

  defaultSimilarityMetric = ModelSimilarityMetrics.pctOverlapOfInput
  defaultModelType = ImbuModelTypes.HTMNetwork

  def __init__(self, cacheRoot, dataPath, loadPath, savePath,
      modelSimilarityMetric):
    self.cacheRoot = cacheRoot
    self.modelSimilarityMetric = (
      modelSimilarityMetric or self.defaultSimilarityMetric
    )
    self.loadPath = loadPath
    self.savePath = savePath
    self.dataPath = dataPath
    self.dataDict = self._loadTrainingData()

  def __repr__(self):
    return ("Imbu<cacheRoot={cacheRoot}, dataPath={dataPath}, "
            "loadPath={loadPath}, savePath={savePath}, "
            "modelSimilarityMetric={modelSimilarityMetric}>"
            .format(**self.__dict__))

  def _defaultModelFactoryKwargs(self):
    return dict(
      numLabels=len(self.dataDict),
      modelDir=self.savePath,
      classifierMetric=self.modelSimilarityMetric.value)

  def modelFactory(self, modelType, **kwargs):

    modelType = modelType or self.defaultModelType

    if modelType is ImbuModelTypes.CioWordFingerprint:
      model = modelType.value(retina=kwargs.get("retina"),
                              apiKey=kwargs.get("apiKey"),
                              fingerprintType=EncoderTypes.word,
                              cacheRoot=self.cacheRoot,
                              **self._defaultModelFactoryKwargs())

    elif modelType is ImbuModelTypes.CioDocumentFingerprint:
      model = modelType.value(retina=kwargs.get("retina"),
                              apiKey=kwargs.get("apiKey"),
                              fingerprintType=EncoderTypes.document,
                              cacheRoot=self.cacheRoot,
                              **self._defaultModelFactoryKwargs())

    elif modelType is ImbuModelTypes.HTMNetwork:
      raise NotImplementedError("HTMNetwork model type is not implemented.")

    else:
      if modelType not in ImbuModelTypes:
        raise NotImplementedError()

      model = modelType.value(**self._defaultModelFactoryKwargs())

    model.verbosity = 0

    return model

  def loadModel(self, *modelFactoryArgs, **modelFactoryKwargs):

    try:
      model = ClassificationModel.load(self.loadPath)
    except IOError as exc:
      model = self.modelFactory(*modelFactoryArgs, **modelFactoryKwargs)
      self.train(model)

    return model

  def _loadTrainingData(self):
    return readCSV(self.dataPath, numLabels=0)

  def train(self, model):
    for seqId, (text, _, _) in enumerate(self.dataDict.values()):
      model.trainDocument(text, [seqId], seqId)

    self.save(model)

  def query(self, model, query, returnDetailedResults=True, sortResults=True):
    return model.inferDocument(query,
                               returnDetailedResults=returnDetailedResults,
                               sortResults=sortResults)

  def save(self, model):
    model.save(self.savePath)

def main():
  parser = argparse.ArgumentParser()

  parser.add_argument("--cacheRoot",
                      help="Root directory in which to cache encodings")
  parser.add_argument("--modelSimilarityMetric",
                      choices={metric.name
                               for metric in ModelSimilarityMetrics},
                      default=ImbuModels.defaultSimilarityMetric.name,
                      help="Classifier metric")
  parser.add_argument("--dataPath",
                      help="Path to data CSV; samples must be in column w/ "
                           "header 'Sample'; see readCSV() for more.",
                      required=True)
  parser.add_argument("--modelName",
                      choices={modelType.name for modelType in ImbuModelTypes},
                      default=ImbuModels.defaultModelType.name,
                      type=str,
                      help="Name of model class. Also used for model results "
                           "directory and pickle checkpoint.")
  parser.add_argument("--loadPath",
                      default="",
                      type=str,
                      help="Path from which to load a serialized model.")
  parser.add_argument("--savePath",
                      default="",
                      type=str,
                      help="Path to save the serialized model.")
  args = parser.parse_args()

  imbu = ImbuModels(
    cacheRoot=args.cacheRoot,
    modelSimilarityMetric=getattr(ModelSimilarityMetrics,
                                  args.modelSimilarityMetric),
    dataPath=args.dataPath,
    loadPath=args.loadPath,
    savePath=args.savePath
  )

  model = imbu.loadModel(getattr(ImbuModelTypes, args.modelName),
                         retina=os.environ.get("IMBU_RETINA_ID"),
                         apiKey=os.environ.get("CORTICAL_API_KEY"))

  # Query the model.
  printTemplate = "{0:<10}|{1:<30}"
  while True:
    print "Now we query the model for samples (quit with 'q')..."

    query = raw_input("Enter a query: ")

    if query == "q":
      break

    _, idList, sortedDistances = imbu.query(model, query)

    print printTemplate.format("Sample ID", "Distance from query")
    for sID, dist in zip(idList, sortedDistances):
      print printTemplate.format(sID, dist)

if __name__ == "__main__":
  main()
<|endoftext|>


<commit_before>import time
import errno
import threading

__all__ = ["lock_file"]

try:
    import fcntl
except ImportError:
    try:
        import msvcrt
    except ImportError:
        raise ImportError("Platform not supported (failed to import fcntl, msvcrt)")
    else:
        _lock_file_blocking_available = False

        def _lock_file_non_blocking(file_):
            try:
                msvcrt.locking(file_.fileno(), msvcrt.LK_NBLCK, 1)
                return True
            # TODO: check errno
            except IOError:
                return False

        def _unlock_file(file_):
            msvcrt.locking(file_.fileno(), msvcrt.LK_UNLCK, 1)

else:
    _lock_file_blocking_available = True
    def _lock_file_blocking(file_):
        fcntl.flock(file_.fileno(), fcntl.LOCK_EX)

    def _lock_file_non_blocking(file_):
        try:
            fcntl.flock(file_.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
            return True
        except IOError as error:
            if error.errno in [errno.EACCES, errno.EAGAIN]:
                return False
            else:
                raise

    def _unlock_file(file_):
        fcntl.flock(file_.fileno(), fcntl.LOCK_UN)

def lock_file(*args, **kwargs):
    return _LockFile(*args, **kwargs)

class LockError(Exception):
    pass

class _LockFile(object):
    def __init__(self, path, timeout=None, retry_period=0.05):
        self._path = path
        self._timeout = timeout
        self._retry_period = retry_period
        self._file = None
        self._thread_lock = threading.Lock()

    def acquire(self):
        self._thread_lock.acquire()
        try:
            if self._file is None:
                self._file = open(self._path, "w")
            if self._timeout is None and _lock_file_blocking_available:
                _lock_file_blocking(self._file)
            else:
                start_time = time.time()
                while True:
                    success = _lock_file_non_blocking(self._file)
                    if success:
                        return
                    elif (self._timeout is not None and
                            time.time() - start_time > self._timeout):
                        raise LockError("Couldn't lock {0}".format(self._path))
                    else:
                        time.sleep(self._retry_period)
        except Exception:
            self._thread_lock.release()
            raise

    def release(self):
        try:
            _unlock_file(self._file)
            self._file.close()
            self._file = None
        finally:
            self._thread_lock.release()

    def __enter__(self):
        self.acquire()
        return self

    def __exit__(self, *args):
        self.release()
        return
<commit_msg>Extract non-blocking acquire code<commit_after>import time
import errno
import threading

__all__ = ["lock_file"]

try:
    import fcntl
except ImportError:
    try:
        import msvcrt
    except ImportError:
        raise ImportError("Platform not supported (failed to import fcntl, msvcrt)")
    else:
        _lock_file_blocking_available = False

        def _lock_file_non_blocking(file_):
            try:
                msvcrt.locking(file_.fileno(), msvcrt.LK_NBLCK, 1)
                return True
            # TODO: check errno
            except IOError:
                return False

        def _unlock_file(file_):
            msvcrt.locking(file_.fileno(), msvcrt.LK_UNLCK, 1)

else:
    _lock_file_blocking_available = True
    def _lock_file_blocking(file_):
        fcntl.flock(file_.fileno(), fcntl.LOCK_EX)

    def _lock_file_non_blocking(file_):
        try:
            fcntl.flock(file_.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
            return True
        except IOError as error:
            if error.errno in [errno.EACCES, errno.EAGAIN]:
                return False
            else:
                raise

    def _unlock_file(file_):
        fcntl.flock(file_.fileno(), fcntl.LOCK_UN)

def lock_file(*args, **kwargs):
    return _LockFile(*args, **kwargs)

class LockError(Exception):
    pass

def _acquire_non_blocking(acquire, timeout, retry_period, error_message):
    start_time = time.time()
    while True:
        success = acquire()
        if success:
            return
        elif (timeout is not None and
                time.time() - start_time > timeout):
            raise LockError(error_message)
        else:
            time.sleep(retry_period)

class _LockFile(object):
    def __init__(self, path, timeout=None, retry_period=0.05):
        self._path = path
        self._timeout = timeout
        self._retry_period = retry_period
        self._file = None
        self._thread_lock = threading.Lock()

    def acquire(self):
        self._thread_lock.acquire()
        try:
            if self._file is None:
                self._file = open(self._path, "w")
            if self._timeout is None and _lock_file_blocking_available:
                _lock_file_blocking(self._file)
            else:
                _acquire_non_blocking(
                    acquire=lambda: _lock_file_non_blocking(self._file),
                    timeout=self._timeout,
                    retry_period=self._retry_period,
                    error_message="Couldn't lock {0}".format(self._path),
                )
        except Exception:
            self._thread_lock.release()
            raise

    def release(self):
        try:
            _unlock_file(self._file)
            self._file.close()
            self._file = None
        finally:
            self._thread_lock.release()

    def __enter__(self):
        self.acquire()
        return self

    def __exit__(self, *args):
        self.release()
        return
<|endoftext|>


<commit_before>from filedes.test.base import BaseFDTestCase
from filedes.socket.unix import make_unix_stream_socket, connect_unix_stream_socket
import unittest2
import _ancillary
import os
import warnings
import time
import threading
import multiprocessing

class TestAncillary(BaseFDTestCase):
    def tempnam(self):
        with warnings.catch_warnings():
            warnings.simplefilter("ignore")
            return os.tempnam(None, 'uxs')

    def testSendRecvFDs(self):
        SN = self.tempnam()

        def worker():
            pid = os.getpid()
            while True:
                try:
                    f1 = connect_unix_stream_socket(SN)
                    break
                except:
                    time.sleep(0.05)
            r, w = _ancillary.recv_fds(f1.fileno(), 2)
            os.write(w, "OK:%d" % pid)
            os.close(r)
            os.close(w)

        def acceptor(r, w):
            f0 = make_unix_stream_socket(SN)
            f0.listen(1024)

            # Accept a single connection, then we're done
            conn, address = f0.accept()
            _ancillary.send_fds(conn.fileno(), [r, w])

        p1 = multiprocessing.Process(target=worker)
        p1.start()

        r, w = os.pipe()
        t = threading.Thread(target=acceptor, args=(r, w))
        t.setDaemon(True)
        t.start()

        msg = os.read(r, 1024)
        msg, pid = msg.split(":")
        self.assertEquals(msg, "OK")
        self.assertEquals(int(pid), p1.pid)
        self.assertNotEquals(int(pid), os.getpid())

        os.close(r)
        os.close(w)

if __name__ == '__main__':
    unittest2.main()
<commit_msg>Some more strictness in ancillary tests<commit_after>from filedes.test.base import BaseFDTestCase
from filedes.socket.unix import make_unix_stream_socket, connect_unix_stream_socket
import unittest2
import _ancillary
import os
import warnings
import time
import threading
import multiprocessing

class TestAncillary(BaseFDTestCase):
    def tempnam(self):
        with warnings.catch_warnings():
            warnings.simplefilter("ignore")
            return os.tempnam(None, 'uxs')

    def testSendRecvFDs(self):
        SN = self.tempnam()

        def worker():
            pid = os.getpid()
            while True:
                try:
                    f1 = connect_unix_stream_socket(SN)
                    break
                except:
                    time.sleep(0.05)
            r, w = _ancillary.recv_fds(f1.fileno(), 2)
            os.write(w, "OK:%d" % pid)
            os.close(r)
            os.close(w)

        def acceptor(r, w):
            f0 = make_unix_stream_socket(SN)
            f0.listen(1024)

            # Accept a single connection, then we're done
            conn, address = f0.accept()
            _ancillary.send_fds(conn.fileno(), [r, w])

        p1 = multiprocessing.Process(target=worker)
        p1.start()

        r, w = os.pipe()
        t = threading.Thread(target=acceptor, args=(r, w))
        t.setDaemon(True)
        t.start()

        msg = os.read(r, 1024)
        msg, pid = msg.split(":")
        self.assertEquals(msg, "OK")
        self.assertEquals(int(pid), p1.pid)
        self.assertNotEquals(int(pid), os.getpid())

        os.close(r)
        os.close(w)

        p1.join()
        self.assertEquals(p1.exitcode, 0)

        t.join()

if __name__ == '__main__':
    unittest2.main()
<|endoftext|>