<commit_before>"""
PHYLIP multiple sequence alignment format (:mod:`skbio.io.format.phylip`)
=========================================================================

.. currentmodule:: skbio.io.format.phylip

The PHYLIP file format stores a multiple sequence alignment. The format was
originally defined and used in Joe Felsenstein's PHYLIP package [1]_, and has
since been supported by several other bioinformatics tools (e.g., RAxML [2]_).
See [3]_ for the original format description, and [4]_ and [5]_ for additional
descriptions.

An example PHYLIP-formatted file taken from [3]_::

          5    42
    Turkey    AAGCTNGGGC ATTTCAGGGT GAGCCCGGGC AATACAGGGT AT
    Salmo gairAAGCCTTGGC AGTGCAGGGT GAGCCGTGGC CGGGCACGGT AT
    H. SapiensACCGGTTGGC CGTTCAGGGT ACAGGTTGGC CGTTCAGGGT AA
    Chimp     AAACCCTTGC CGTTACGCTT AAACCGAGGC CGGGACACTC AT
    Gorilla   AAACCCTTGC CGGTACGCTT AAACCATTGC CGGTACGCTT AA

.. note:: Original copyright notice for the above PHYLIP file:

   *(c) Copyright 1986-2008 by The University of Washington. Written by Joseph
   Felsenstein. Permission is granted to copy this document provided that no
   fee is charged for it and that this copyright notice is not removed.*

Format Support
--------------
**Has Sniffer: Yes**

+------+------+---------------------------------------------------------------+
|Reader|Writer|                          Object Class                         |
+======+======+===============================================================+
|Yes   |Yes   |:mod:`skbio.alignment.Alignment`                               |
+------+------+---------------------------------------------------------------+

Format Specification
--------------------
PHYLIP format is a plain text format containing exactly two sections: a header
describing the dimensions of the alignment, followed by the multiple sequence
alignment itself.

The format described here is "strict" PHYLIP, as described in [4]_. Strict
PHYLIP requires that each sequence identifier is exactly 10 characters long
(padded with spaces as necessary). Other bioinformatics tools (e.g., RAxML) may
relax this rule to allow for longer sequence identifiers. See the
**Alignment Section** below for more details.

The format described here is "sequential" format. The original PHYLIP format
specification [3]_ describes both sequential and interleaved formats.

.. note:: scikit-bio currently only supports writing strict, sequential
   PHYLIP-formatted files from an ``skbio.alignment.Alignment``. It does not
   yet support reading PHYLIP-formatted files, nor does it support relaxed or
   interleaved PHYLIP formats.

Header Section
^^^^^^^^^^^^^^
The header consists of a single line describing the dimensions of the
alignment. It **must** be the first line in the file. The header consists of
optional spaces, followed by two positive integers (``n`` and ``m``) separated
by one or more spaces. The first integer (``n``) specifies the number of
sequences (i.e., the number of rows) in the alignment. The second integer
(``m``) specifies the length of the sequences (i.e., the number of columns) in
the alignment. The smallest supported alignment dimensions are 1x1.

.. note:: scikit-bio will write the PHYLIP format header *without* preceding
   spaces, and with only a single space between ``n`` and ``m``.

   PHYLIP format *does not* support blank line(s) between the header and the
   alignment.

Alignment Section
^^^^^^^^^^^^^^^^^
The alignment section immediately follows the header. It consists of ``n``
lines (rows), one for each sequence in the alignment. Each row consists of a
sequence identifier (ID) and characters in the sequence, in fixed width format.

The sequence ID can be up to 10 characters long. IDs less than 10 characters
must have spaces appended to them to reach the 10 character fixed width. Within
an ID, all characters except newlines are supported, including spaces,
underscores, and numbers.

.. note:: While not explicitly stated in the original PHYLIP format
   description, scikit-bio only supports writing unique sequence identifiers
   (i.e., duplicates are not allowed). Uniqueness is required because an
   ``skbio.alignment.Alignment`` cannot be created with duplicate IDs.

   scikit-bio supports the empty string (``''``) as a valid sequence ID. An
   empty ID will be padded with 10 spaces.

Sequence characters immediately follow the sequence ID. They *must* start at
the 11th character in the line, as the first 10 characters are reserved for the
sequence ID. While PHYLIP format does not explicitly restrict the set of
supported characters that may be used to represent a sequence, the original
format description [3]_ specifies the IUPAC nucleic acid lexicon for DNA or RNA
sequences, and the IUPAC protein lexicon for protein sequences. The original
PHYLIP specification uses ``-`` as a gap character, though older versions also
supported ``.``. The sequence characters may contain optional spaces (e.g., to
improve readability), and both upper and lower case characters are supported.

.. note:: scikit-bio will write a PHYLIP-formatted file even if the alignment's
   sequence characters are not valid IUPAC characters. This differs from the
   PHYLIP specification, which states that a PHYLIP-formatted file can only
   contain valid IUPAC characters. To check whether all characters are valid
   before writing, the user can call ``Alignment.is_valid()``.

   Since scikit-bio supports both ``-`` and ``.`` as gap characters (e.g., in
   ``skbio.alignment.Alignment``), both are supported when writing a
   PHYLIP-formatted file.

   When writing a PHYLIP-formatted file, scikit-bio will split up each sequence
   into chunks that are 10 characters long. Each chunk will be separated by a
   single space. The sequence will always appear on a single line (sequential
   format). It will *not* be wrapped across multiple lines. Sequences are
   chunked in this manner for improved readability, and because most example
   PHYLIP files are chunked in a similar way (e.g., see the example file
   above). Note that this chunking is not required by the PHYLIP format.

Examples
--------
Let's create an alignment with three DNA sequences of equal length:

>>> from skbio import Alignment, DNA
>>> seqs = [DNA('ACCGTTGTA-GTAGCT', metadata={'id':'seq1'}),
...         DNA('A--GTCGAA-GTACCT', metadata={'id':'sequence-2'}),
...         DNA('AGAGTTGAAGGTATCT', metadata={'id':'3'})]
>>> aln = Alignment(seqs)
>>> aln
<Alignment: n=3; mean +/- std length=16.00 +/- 0.00>

Now let's write the alignment to file in PHYLIP format, and take a look at the
output:

>>> from io import StringIO
>>> fh = StringIO()
>>> print(aln.write(fh, format='phylip').getvalue())
3 16
seq1      ACCGTTGTA- GTAGCT
sequence-2A--GTCGAA- GTACCT
3         AGAGTTGAAG GTATCT
<BLANKLINE>
>>> fh.close()

Notice that the 16-character sequences were split into two chunks, and that
each sequence appears on a single line (sequential format). Also note that each
sequence ID is padded with spaces to 10 characters in order to produce a fixed
width column.

If the sequence IDs in an alignment surpass the 10-character limit, an error
will be raised when we try to write a PHYLIP file:

>>> long_id_seqs = [DNA('ACCGT', metadata={'id':'seq1'}),
...                 DNA('A--GT', metadata={'id':'long-sequence-2'}),
...                 DNA('AGAGT', metadata={'id':'seq3'})]
>>> long_id_aln = Alignment(long_id_seqs)
>>> fh = StringIO()
>>> long_id_aln.write(fh, format='phylip')
Traceback (most recent call last):
    ...
PhylipFormatError: Alignment can only be written in PHYLIP format if all \
sequence IDs have 10 or fewer characters. Found sequence with ID \
'long-sequence-2' that exceeds this limit. Use Alignment.update_ids to assign \
shorter IDs.
>>> fh.close()

One way to work around this is to update the IDs to be shorter. The recommended
way of accomplishing this is via ``Alignment.update_ids``, which provides a
flexible way of creating a new ``Alignment`` with updated IDs. For example, to
remap each of the IDs to integer-based IDs:

>>> short_id_aln, _ = long_id_aln.update_ids()
>>> short_id_aln.ids()
['1', '2', '3']

We can now write the new alignment in PHYLIP format:

>>> fh = StringIO()
>>> print(short_id_aln.write(fh, format='phylip').getvalue())
3 5
1         ACCGT
2         A--GT
3         AGAGT
<BLANKLINE>
>>> fh.close()

References
----------
.. [1] http://evolution.genetics.washington.edu/phylip.html
.. [2] RAxML Version 8: A tool for Phylogenetic Analysis and
   Post-Analysis of Large Phylogenies". In Bioinformatics, 2014
.. [3] http://evolution.genetics.washington.edu/phylip/doc/sequence.html
.. [4] http://www.phylo.org/tools/obsolete/phylip.html
.. [5] http://www.bioperl.org/wiki/PHYLIP_multiple_alignment_format

"""

# ----------------------------------------------------------------------------
# Copyright (c) 2013--, scikit-bio development team.
#
# Distributed under the terms of the Modified BSD License.
#
# The full license is in the file COPYING.txt, distributed with this software.
# ----------------------------------------------------------------------------

from __future__ import (absolute_import, division, print_function,
                        unicode_literals)

from skbio.alignment import Alignment
from skbio.sequence import Sequence
from skbio.io import create_format, PhylipFormatError
from skbio.util._misc import chunk_str

phylip = create_format('phylip')

@phylip.sniffer()
def _phylip_sniffer(fh):
    # Strategy:
    #   Read the header and a single sequence; verify that the sequence length
    #   matches the header information.  Do not verify that the total number of
    #   lines matches the header information, since that would require reading
    #   the whole file.
    try:
        header = next(_line_generator(fh))
    except StopIteration:
        return False, {}
    try:
        _, seq_len = _validate_header(header)
    except PhylipFormatError:
        return False, {}
    line = next(_line_generator(fh))
    try:
        _validate_line(line, seq_len)
    except PhylipFormatError:
        return False, {}
    return True, {}

@phylip.reader(Alignment)
def _phylip_to_alignment(fh, constructor=Sequence):
    return Alignment([constructor(seq, metadata={'id': ID})
                      for (seq, ID) in _parse_phylip_raw(fh)])

@phylip.writer(Alignment)
def _alignment_to_phylip(obj, fh):

    if obj.is_empty():
        raise PhylipFormatError(
            "Alignment can only be written in PHYLIP format if there is at "
            "least one sequence in the alignment.")

    sequence_length = obj.sequence_length()
    if sequence_length == 0:
        raise PhylipFormatError(
            "Alignment can only be written in PHYLIP format if there is at "
            "least one position in the alignment.")

    chunk_size = 10
    for id_ in obj.ids():
        if len(id_) > chunk_size:
            raise PhylipFormatError(
                "Alignment can only be written in PHYLIP format if all "
                "sequence IDs have %d or fewer characters. Found sequence "
                "with ID '%s' that exceeds this limit. Use "
                "Alignment.update_ids to assign shorter IDs." %
                (chunk_size, id_))

    sequence_count = obj.sequence_count()
    fh.write('{0:d} {1:d}\n'.format(sequence_count, sequence_length))

    fmt = '{0:%d}{1}\n' % chunk_size
    for seq in obj:
        chunked_seq = chunk_str(str(seq), chunk_size, ' ')
        fh.write(fmt.format(seq.metadata['id'], chunked_seq))

def _validate_header(header):
    header_vals = header.split()
    try:
        n_seqs, seq_len = [int(x) for x in header_vals]
        if n_seqs < 1 or seq_len < 1:
            raise PhylipFormatError(
                'The number of sequences and the length must be positive.')
    except ValueError:
        raise PhylipFormatError(
            'Found non-header line when attempting to read the 1st record '
            '(header line should have two space-separated integers): '
            '"%s"' % header)
    return n_seqs, seq_len

def _validate_line(line, seq_len):
    if not line:
        raise PhylipFormatError("Empty lines are not allowed.")
    ID = line[:10].strip()
    seq = line[10:].replace(' ', '')
    if len(seq) != seq_len:
        raise PhylipFormatError(
            "The length of sequence %s is not %s as specified in the header."
            % (ID, seq_len))
    return (seq, ID)

def _parse_phylip_raw(fh):
    """Raw parser for PHYLIP files.

    Returns a list of raw (seq, id) values.  It is the responsibility of the
    caller to construct the correct in-memory object to hold the data.

    """
    # Note: this returns the full data instead of yielding each sequence,
    # because the header specifies the number of sequences, so the file cannot
    # be validated until it's read completely.

    # File should have a single header on the first line.
    try:
        header = next(_line_generator(fh))
    except StopIteration:
        raise PhylipFormatError("This file is empty.")
    n_seqs, seq_len = _validate_header(header)

    # All following lines should be ID+sequence. No blank lines are allowed.
    data = []
    for line in _line_generator(fh):
        data.append(_validate_line(line, seq_len))
    if len(data) != n_seqs:
        raise PhylipFormatError(
            "The number of sequences is not %s " % n_seqs +
            "as specified in the header.")
    return data

def _line_generator(fh):
    """Just remove linebreak characters and yield lines.
    """
    for line in fh:
        yield line[:-1]
<commit_msg>Collapsed all try/except to one in phylip sniffer<commit_after>"""
PHYLIP multiple sequence alignment format (:mod:`skbio.io.format.phylip`)
=========================================================================

.. currentmodule:: skbio.io.format.phylip

The PHYLIP file format stores a multiple sequence alignment. The format was
originally defined and used in Joe Felsenstein's PHYLIP package [1]_, and has
since been supported by several other bioinformatics tools (e.g., RAxML [2]_).
See [3]_ for the original format description, and [4]_ and [5]_ for additional
descriptions.

An example PHYLIP-formatted file taken from [3]_::

          5    42
    Turkey    AAGCTNGGGC ATTTCAGGGT GAGCCCGGGC AATACAGGGT AT
    Salmo gairAAGCCTTGGC AGTGCAGGGT GAGCCGTGGC CGGGCACGGT AT
    H. SapiensACCGGTTGGC CGTTCAGGGT ACAGGTTGGC CGTTCAGGGT AA
    Chimp     AAACCCTTGC CGTTACGCTT AAACCGAGGC CGGGACACTC AT
    Gorilla   AAACCCTTGC CGGTACGCTT AAACCATTGC CGGTACGCTT AA

.. note:: Original copyright notice for the above PHYLIP file:

   *(c) Copyright 1986-2008 by The University of Washington. Written by Joseph
   Felsenstein. Permission is granted to copy this document provided that no
   fee is charged for it and that this copyright notice is not removed.*

Format Support
--------------
**Has Sniffer: Yes**

+------+------+---------------------------------------------------------------+
|Reader|Writer|                          Object Class                         |
+======+======+===============================================================+
|Yes   |Yes   |:mod:`skbio.alignment.Alignment`                               |
+------+------+---------------------------------------------------------------+

Format Specification
--------------------
PHYLIP format is a plain text format containing exactly two sections: a header
describing the dimensions of the alignment, followed by the multiple sequence
alignment itself.

The format described here is "strict" PHYLIP, as described in [4]_. Strict
PHYLIP requires that each sequence identifier is exactly 10 characters long
(padded with spaces as necessary). Other bioinformatics tools (e.g., RAxML) may
relax this rule to allow for longer sequence identifiers. See the
**Alignment Section** below for more details.

The format described here is "sequential" format. The original PHYLIP format
specification [3]_ describes both sequential and interleaved formats.

.. note:: scikit-bio currently only supports writing strict, sequential
   PHYLIP-formatted files from an ``skbio.alignment.Alignment``. It does not
   yet support reading PHYLIP-formatted files, nor does it support relaxed or
   interleaved PHYLIP formats.

Header Section
^^^^^^^^^^^^^^
The header consists of a single line describing the dimensions of the
alignment. It **must** be the first line in the file. The header consists of
optional spaces, followed by two positive integers (``n`` and ``m``) separated
by one or more spaces. The first integer (``n``) specifies the number of
sequences (i.e., the number of rows) in the alignment. The second integer
(``m``) specifies the length of the sequences (i.e., the number of columns) in
the alignment. The smallest supported alignment dimensions are 1x1.

.. note:: scikit-bio will write the PHYLIP format header *without* preceding
   spaces, and with only a single space between ``n`` and ``m``.

   PHYLIP format *does not* support blank line(s) between the header and the
   alignment.

Alignment Section
^^^^^^^^^^^^^^^^^
The alignment section immediately follows the header. It consists of ``n``
lines (rows), one for each sequence in the alignment. Each row consists of a
sequence identifier (ID) and characters in the sequence, in fixed width format.

The sequence ID can be up to 10 characters long. IDs less than 10 characters
must have spaces appended to them to reach the 10 character fixed width. Within
an ID, all characters except newlines are supported, including spaces,
underscores, and numbers.

.. note:: While not explicitly stated in the original PHYLIP format
   description, scikit-bio only supports writing unique sequence identifiers
   (i.e., duplicates are not allowed). Uniqueness is required because an
   ``skbio.alignment.Alignment`` cannot be created with duplicate IDs.

   scikit-bio supports the empty string (``''``) as a valid sequence ID. An
   empty ID will be padded with 10 spaces.

Sequence characters immediately follow the sequence ID. They *must* start at
the 11th character in the line, as the first 10 characters are reserved for the
sequence ID. While PHYLIP format does not explicitly restrict the set of
supported characters that may be used to represent a sequence, the original
format description [3]_ specifies the IUPAC nucleic acid lexicon for DNA or RNA
sequences, and the IUPAC protein lexicon for protein sequences. The original
PHYLIP specification uses ``-`` as a gap character, though older versions also
supported ``.``. The sequence characters may contain optional spaces (e.g., to
improve readability), and both upper and lower case characters are supported.

.. note:: scikit-bio will write a PHYLIP-formatted file even if the alignment's
   sequence characters are not valid IUPAC characters. This differs from the
   PHYLIP specification, which states that a PHYLIP-formatted file can only
   contain valid IUPAC characters. To check whether all characters are valid
   before writing, the user can call ``Alignment.is_valid()``.

   Since scikit-bio supports both ``-`` and ``.`` as gap characters (e.g., in
   ``skbio.alignment.Alignment``), both are supported when writing a
   PHYLIP-formatted file.

   When writing a PHYLIP-formatted file, scikit-bio will split up each sequence
   into chunks that are 10 characters long. Each chunk will be separated by a
   single space. The sequence will always appear on a single line (sequential
   format). It will *not* be wrapped across multiple lines. Sequences are
   chunked in this manner for improved readability, and because most example
   PHYLIP files are chunked in a similar way (e.g., see the example file
   above). Note that this chunking is not required by the PHYLIP format.

Examples
--------
Let's create an alignment with three DNA sequences of equal length:

>>> from skbio import Alignment, DNA
>>> seqs = [DNA('ACCGTTGTA-GTAGCT', metadata={'id':'seq1'}),
...         DNA('A--GTCGAA-GTACCT', metadata={'id':'sequence-2'}),
...         DNA('AGAGTTGAAGGTATCT', metadata={'id':'3'})]
>>> aln = Alignment(seqs)
>>> aln
<Alignment: n=3; mean +/- std length=16.00 +/- 0.00>

Now let's write the alignment to file in PHYLIP format, and take a look at the
output:

>>> from io import StringIO
>>> fh = StringIO()
>>> print(aln.write(fh, format='phylip').getvalue())
3 16
seq1      ACCGTTGTA- GTAGCT
sequence-2A--GTCGAA- GTACCT
3         AGAGTTGAAG GTATCT
<BLANKLINE>
>>> fh.close()

Notice that the 16-character sequences were split into two chunks, and that
each sequence appears on a single line (sequential format). Also note that each
sequence ID is padded with spaces to 10 characters in order to produce a fixed
width column.

If the sequence IDs in an alignment surpass the 10-character limit, an error
will be raised when we try to write a PHYLIP file:

>>> long_id_seqs = [DNA('ACCGT', metadata={'id':'seq1'}),
...                 DNA('A--GT', metadata={'id':'long-sequence-2'}),
...                 DNA('AGAGT', metadata={'id':'seq3'})]
>>> long_id_aln = Alignment(long_id_seqs)
>>> fh = StringIO()
>>> long_id_aln.write(fh, format='phylip')
Traceback (most recent call last):
    ...
PhylipFormatError: Alignment can only be written in PHYLIP format if all \
sequence IDs have 10 or fewer characters. Found sequence with ID \
'long-sequence-2' that exceeds this limit. Use Alignment.update_ids to assign \
shorter IDs.
>>> fh.close()

One way to work around this is to update the IDs to be shorter. The recommended
way of accomplishing this is via ``Alignment.update_ids``, which provides a
flexible way of creating a new ``Alignment`` with updated IDs. For example, to
remap each of the IDs to integer-based IDs:

>>> short_id_aln, _ = long_id_aln.update_ids()
>>> short_id_aln.ids()
['1', '2', '3']

We can now write the new alignment in PHYLIP format:

>>> fh = StringIO()
>>> print(short_id_aln.write(fh, format='phylip').getvalue())
3 5
1         ACCGT
2         A--GT
3         AGAGT
<BLANKLINE>
>>> fh.close()

References
----------
.. [1] http://evolution.genetics.washington.edu/phylip.html
.. [2] RAxML Version 8: A tool for Phylogenetic Analysis and
   Post-Analysis of Large Phylogenies". In Bioinformatics, 2014
.. [3] http://evolution.genetics.washington.edu/phylip/doc/sequence.html
.. [4] http://www.phylo.org/tools/obsolete/phylip.html
.. [5] http://www.bioperl.org/wiki/PHYLIP_multiple_alignment_format

"""

# ----------------------------------------------------------------------------
# Copyright (c) 2013--, scikit-bio development team.
#
# Distributed under the terms of the Modified BSD License.
#
# The full license is in the file COPYING.txt, distributed with this software.
# ----------------------------------------------------------------------------

from __future__ import (absolute_import, division, print_function,
                        unicode_literals)

from skbio.alignment import Alignment
from skbio.sequence import Sequence
from skbio.io import create_format, PhylipFormatError
from skbio.util._misc import chunk_str

phylip = create_format('phylip')

@phylip.sniffer()
def _phylip_sniffer(fh):
    # Strategy:
    #   Read the header and a single sequence; verify that the sequence length
    #   matches the header information.  Do not verify that the total number of
    #   lines matches the header information, since that would require reading
    #   the whole file.
    try:
        header = next(_line_generator(fh))
        _, seq_len = _validate_header(header)
        line = next(_line_generator(fh))
        _validate_line(line, seq_len)
    except (StopIteration, PhylipFormatError):
        return False, {}
    return True, {}

@phylip.reader(Alignment)
def _phylip_to_alignment(fh, constructor=Sequence):
    return Alignment([constructor(seq, metadata={'id': ID})
                      for (seq, ID) in _parse_phylip_raw(fh)])

@phylip.writer(Alignment)
def _alignment_to_phylip(obj, fh):

    if obj.is_empty():
        raise PhylipFormatError(
            "Alignment can only be written in PHYLIP format if there is at "
            "least one sequence in the alignment.")

    sequence_length = obj.sequence_length()
    if sequence_length == 0:
        raise PhylipFormatError(
            "Alignment can only be written in PHYLIP format if there is at "
            "least one position in the alignment.")

    chunk_size = 10
    for id_ in obj.ids():
        if len(id_) > chunk_size:
            raise PhylipFormatError(
                "Alignment can only be written in PHYLIP format if all "
                "sequence IDs have %d or fewer characters. Found sequence "
                "with ID '%s' that exceeds this limit. Use "
                "Alignment.update_ids to assign shorter IDs." %
                (chunk_size, id_))

    sequence_count = obj.sequence_count()
    fh.write('{0:d} {1:d}\n'.format(sequence_count, sequence_length))

    fmt = '{0:%d}{1}\n' % chunk_size
    for seq in obj:
        chunked_seq = chunk_str(str(seq), chunk_size, ' ')
        fh.write(fmt.format(seq.metadata['id'], chunked_seq))

def _validate_header(header):
    header_vals = header.split()
    try:
        n_seqs, seq_len = [int(x) for x in header_vals]
        if n_seqs < 1 or seq_len < 1:
            raise PhylipFormatError(
                'The number of sequences and the length must be positive.')
    except ValueError:
        raise PhylipFormatError(
            'Found non-header line when attempting to read the 1st record '
            '(header line should have two space-separated integers): '
            '"%s"' % header)
    return n_seqs, seq_len

def _validate_line(line, seq_len):
    if not line:
        raise PhylipFormatError("Empty lines are not allowed.")
    ID = line[:10].strip()
    seq = line[10:].replace(' ', '')
    if len(seq) != seq_len:
        raise PhylipFormatError(
            "The length of sequence %s is not %s as specified in the header."
            % (ID, seq_len))
    return (seq, ID)

def _parse_phylip_raw(fh):
    """Raw parser for PHYLIP files.

    Returns a list of raw (seq, id) values.  It is the responsibility of the
    caller to construct the correct in-memory object to hold the data.

    """
    # Note: this returns the full data instead of yielding each sequence,
    # because the header specifies the number of sequences, so the file cannot
    # be validated until it's read completely.

    # File should have a single header on the first line.
    try:
        header = next(_line_generator(fh))
    except StopIteration:
        raise PhylipFormatError("This file is empty.")
    n_seqs, seq_len = _validate_header(header)

    # All following lines should be ID+sequence. No blank lines are allowed.
    data = []
    for line in _line_generator(fh):
        data.append(_validate_line(line, seq_len))
    if len(data) != n_seqs:
        raise PhylipFormatError(
            "The number of sequences is not %s " % n_seqs +
            "as specified in the header.")
    return data

def _line_generator(fh):
    """Just remove linebreak characters and yield lines.
    """
    for line in fh:
        yield line[:-1]
<|endoftext|>


<commit_before># Licensed under a 3-clause BSD style license, see LICENSE.
"""
Submodule for helpers to the Dataset-like classes
=================================================

User-facing classes that structure the provenance information, i.e. the history of operations performed on the dataset.

**Available classes:**

* ``Provenance`` (abstract base class).
* ``Origin`` (abstract base class).
* ``ObjectOrigin``.
* ``FileOrigin``.
* ``Transformation``.
* ``Formatting``.
"""

# -----------------------------------------------------------------------------
# Import statements
# -----------------------------------------------------------------------------
from __future__ import absolute_import

import json

from .exceptions import SkhepTypeError
from .py23 import *

import pytest

# -----------------------------------------------------------------------------
# Provenance-like classes
# -----------------------------------------------------------------------------
class Provenance(object):
    """
    Abstract base class for all classes containing provenance information.

    Trying to instantiate it raises an exception. Instantiate one of its subclasses instead.
    """

    def __init__(self):
        raise SkhepTypeError('Provenance')

    @property
    def detail(self):
        """
        String providing detailed information about the origin, transformation, or formatting.
        """
        raise NotImplementedError

class Origin(Provenance):
    """
    Abstract base class for all classes describing the first object in a provenance list.

    Trying to instantiate it raises an exception. Instantiate one of its subclasses instead.
    """

    def __init__(self):
        raise SkhepTypeError('Origin')

class ObjectOrigin(Origin):
    """
    Declares that the dataset came from some Python object.
    Its history prior to that is unknown.

    :Parameters:
    detail: str
        String providing detailed information about the object origin.

    :Examples:
    >>> from skhep.utils import ObjectOrigin
    >>> from array import array
    >>> data = array('i',[1,2,3])
    >>> provenance1 = ObjectOrigin(repr(data))
    >>> provenance1
    <ObjectOrigin>
    >>> provenance1.detail
    "array('i', [1, 2, 3])"

    >>> provenance2 = ObjectOrigin('array_of_ints')
    >>> provenance2.detail
    'array_of_ints'
    """

    def __init__(self, detail):
        if not isinstance(detail, string_types):
            assert False, 'Argument is not of string type!'
        self._detail = detail

    @property
    def detail(self):
        return self._detail

    def __repr__(self):
        return "<ObjectOrigin>"

class FileOrigin(Origin):
    """
    Declares that the dataset came from a file or a set of files.

    :Parameters:
    files: str or iterable of str or file objects
        File name(s) or object(s).

    :Examples:
    >>> from skhep.utils import FileOrigin
    >>> prov = FileOrigin(['file1.root', 'file2.root','file3.root'])
    >>> prov
    <FileOrigin (3 files)>
    """

    def __init__(self, files):
        if isinstance(files, string_types):
            self.files = (files,)
        else:
            self.files = []
            for x in files:
                if isinstance(x, file):
                    self.files.append(x.name)
                elif isinstance(x, string_types):
                    self.files.append(x)
                else:
                    assert False, ('Argument must be a string filename, an iterable of file objects, '
                                   'or an iterable of string filenames!')
            self.files = tuple(self.files)

    @property
    def detail(self):
        return ", ".join(json.dumps(x) for x in self.files)

    def __repr__(self):
        return "<FileOrigin ({0} file{1})>".format(len(self.files),'s' if len(self.files)>1 else '')

class Transformation(Provenance):
    """
    Declares that the dataset was transformed by some mathematical operation.

    :Parameters:
    name: str
        String detailing how the dataset got transformed.
    args: iterable, optional
        Optional set of arguments given extra detail on the transformation.

    :Examples:
    >>> from skhep.utils import Transformation
    >>> transf = Transformation('all elms * 2')
    >>> transf
    <Transformation(all elms * 2)>
    """

    def __init__(self, name, args=[]):
        self.name = name
        self.args = args

    @property
    def detail(self):
        return "{0} ({1})".format(self.name,
                                 ", ".format(x.detail if isinstance(x, Provenance) else repr(x) for x in self.args))

    def __repr__(self):
        return "<Transformation({0})>".format(self.name)

class Formatting(Provenance):
    """
    Declares that the dataset was reformatted, keeping its semantic meaning, but changed in representation.
    """

    def __init__(self, format, args):
        self.format = format
        self.args = args

    @property
    def detail(self):
        return "{0}({1})".format(self.format,
                                 ", ".format(x.detail if isinstance(x, Provenance) else repr(x) for x in self.args))
<commit_msg>Removed unnecessary dependency (import statement).<commit_after># Licensed under a 3-clause BSD style license, see LICENSE.
"""
Submodule for helpers to the Dataset-like classes
=================================================

User-facing classes that structure the provenance information, i.e. the history of operations performed on the dataset.

**Available classes:**

* ``Provenance`` (abstract base class).
* ``Origin`` (abstract base class).
* ``ObjectOrigin``.
* ``FileOrigin``.
* ``Transformation``.
* ``Formatting``.
"""

# -----------------------------------------------------------------------------
# Import statements
# -----------------------------------------------------------------------------
from __future__ import absolute_import

import json

from .exceptions import SkhepTypeError
from .py23 import *

# -----------------------------------------------------------------------------
# Provenance-like classes
# -----------------------------------------------------------------------------
class Provenance(object):
    """
    Abstract base class for all classes containing provenance information.

    Trying to instantiate it raises an exception. Instantiate one of its subclasses instead.
    """

    def __init__(self):
        raise SkhepTypeError('Provenance')

    @property
    def detail(self):
        """
        String providing detailed information about the origin, transformation, or formatting.
        """
        raise NotImplementedError

class Origin(Provenance):
    """
    Abstract base class for all classes describing the first object in a provenance list.

    Trying to instantiate it raises an exception. Instantiate one of its subclasses instead.
    """

    def __init__(self):
        raise SkhepTypeError('Origin')

class ObjectOrigin(Origin):
    """
    Declares that the dataset came from some Python object.
    Its history prior to that is unknown.

    :Parameters:
    detail: str
        String providing detailed information about the object origin.

    :Examples:
    >>> from skhep.utils import ObjectOrigin
    >>> from array import array
    >>> data = array('i',[1,2,3])
    >>> provenance1 = ObjectOrigin(repr(data))
    >>> provenance1
    <ObjectOrigin>
    >>> provenance1.detail
    "array('i', [1, 2, 3])"

    >>> provenance2 = ObjectOrigin('array_of_ints')
    >>> provenance2.detail
    'array_of_ints'
    """

    def __init__(self, detail):
        if not isinstance(detail, string_types):
            assert False, 'Argument is not of string type!'
        self._detail = detail

    @property
    def detail(self):
        return self._detail

    def __repr__(self):
        return "<ObjectOrigin>"

class FileOrigin(Origin):
    """
    Declares that the dataset came from a file or a set of files.

    :Parameters:
    files: str or iterable of str or file objects
        File name(s) or object(s).

    :Examples:
    >>> from skhep.utils import FileOrigin
    >>> prov = FileOrigin(['file1.root', 'file2.root','file3.root'])
    >>> prov
    <FileOrigin (3 files)>
    """

    def __init__(self, files):
        if isinstance(files, string_types):
            self.files = (files,)
        else:
            self.files = []
            for x in files:
                if isinstance(x, file):
                    self.files.append(x.name)
                elif isinstance(x, string_types):
                    self.files.append(x)
                else:
                    assert False, ('Argument must be a string filename, an iterable of file objects, '
                                   'or an iterable of string filenames!')
            self.files = tuple(self.files)

    @property
    def detail(self):
        return ", ".join(json.dumps(x) for x in self.files)

    def __repr__(self):
        return "<FileOrigin ({0} file{1})>".format(len(self.files),'s' if len(self.files)>1 else '')

class Transformation(Provenance):
    """
    Declares that the dataset was transformed by some mathematical operation.

    :Parameters:
    name: str
        String detailing how the dataset got transformed.
    args: iterable, optional
        Optional set of arguments given extra detail on the transformation.

    :Examples:
    >>> from skhep.utils import Transformation
    >>> transf = Transformation('all elms * 2')
    >>> transf
    <Transformation(all elms * 2)>
    """

    def __init__(self, name, args=[]):
        self.name = name
        self.args = args

    @property
    def detail(self):
        return "{0} ({1})".format(self.name,
                                 ", ".format(x.detail if isinstance(x, Provenance) else repr(x) for x in self.args))

    def __repr__(self):
        return "<Transformation({0})>".format(self.name)

class Formatting(Provenance):
    """
    Declares that the dataset was reformatted, keeping its semantic meaning, but changed in representation.
    """

    def __init__(self, format, args):
        self.format = format
        self.args = args

    @property
    def detail(self):
        return "{0}({1})".format(self.format,
                                 ", ".format(x.detail if isinstance(x, Provenance) else repr(x) for x in self.args))
<|endoftext|>


<commit_before>"""
2015. 12. 10
Hans Roh
"""
from . import Saddle, part

Saddle = Saddle.Saddle
Saddlery = part.Part

request_started = 0
request_finished = 1
request_exception_occured = 2
request_tearing_down = 3
template_rendered = 4
message_flashed = 5

<commit_msg>remove saddle.Saddlery<commit_after>"""
2015. 12. 10
Hans Roh
"""
from . import Saddle, part

Saddle = Saddle.Saddle

# deprecated in version 0.26.11
#Saddlery = part.Part

request_started = 0
request_finished = 1
request_exception_occured = 2
request_tearing_down = 3
template_rendered = 4
message_flashed = 5

<|endoftext|>


<commit_before># -*- coding: utf-8 -*-
##
## $Id$
##
## This file is part of CDS Invenio.
## Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 CERN.
##
## CDS Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## CDS Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with CDS Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.

"""WebAlert Regression Test Suite."""

__revision__ = "$Id$"

import unittest

from invenio.config import CFG_SITE_URL
from invenio.testutils import make_test_suite, run_test_suite, \
                              test_web_page_content, merge_error_messages
from invenio.htmlparser import get_as_text

class WebAlertWebPagesAvailabilityTest(unittest.TestCase):
    """Check WebAlert web pages whether they are up or not."""

    def test_your_alerts_pages_availability(self):
        """webalert - availability of Your Alerts pages"""

        baseurl = CFG_SITE_URL + '/youralerts/'

        _exports = ['', 'display', 'input', 'modify', 'list', 'add',
                    'update', 'remove']

        error_messages = []
        for url in [baseurl + page for page in _exports]:
            error_messages.extend(test_web_page_content(url))
        if error_messages:
            self.fail(merge_error_messages(error_messages))
        return

class WebAlertHTMLToTextTest(unittest.TestCase):
    """Check that HTML is properly converted to text."""

    def test_your_alerts_pages_availability(self):
        """webalert - HTML to text conversion"""

        self.assertEqual("High energy cosmic rays striking atoms at the top of the atmosphere give the rise to showers of particles striking the Earth's surface \nDes rayons cosmiques de haute energie heurtent des atomes dans la haute atmosphere et donnent ainsi naissance a des gerbes de particules projetees sur la surface terrestre  \n 10 May 1999        \nPicture number: CERN-DI-9905005", get_as_text(5))

        self.assertEqual("Introducion to Nuclear Physics: course; 2006 ed.  / Chomaz, P      \nAtomic nuclei are made of nucleons, protons and neutrons, composed by quarks strongly interacting via gluons. [...]   \nExternal links: Fulltext : <Talk 06 Jul 2006>; Fulltext : <Talk 06 Jul 2006>; Fulltext : <Talk 07 Jul 2006>; Fulltext : <Talk 07 Jul 2006>; Fulltext : <Talk 10 Jul 2006>; Fulltext : <Talk 10 Jul 2006>; Fulltext : <Talk 11 Jul 2006>; Fulltext : <More information>", get_as_text(97))

TEST_SUITE = make_test_suite(WebAlertWebPagesAvailabilityTest,
                             WebAlertHTMLToTextTest)

if __name__ == "__main__":
    run_test_suite(TEST_SUITE, warn_user=True)
<commit_msg>Fixed regression test.<commit_after># -*- coding: utf-8 -*-
##
## $Id$
##
## This file is part of CDS Invenio.
## Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 CERN.
##
## CDS Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## CDS Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with CDS Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.

"""WebAlert Regression Test Suite."""

__revision__ = "$Id$"

import unittest

from invenio.config import CFG_SITE_URL
from invenio.testutils import make_test_suite, run_test_suite, \
                              test_web_page_content, merge_error_messages
from invenio.htmlparser import get_as_text

class WebAlertWebPagesAvailabilityTest(unittest.TestCase):
    """Check WebAlert web pages whether they are up or not."""

    def test_your_alerts_pages_availability(self):
        """webalert - availability of Your Alerts pages"""

        baseurl = CFG_SITE_URL + '/youralerts/'

        _exports = ['', 'display', 'input', 'modify', 'list', 'add',
                    'update', 'remove']

        error_messages = []
        for url in [baseurl + page for page in _exports]:
            error_messages.extend(test_web_page_content(url))
        if error_messages:
            self.fail(merge_error_messages(error_messages))
        return

class WebAlertHTMLToTextTest(unittest.TestCase):
    """Check that HTML is properly converted to text."""

    def test_your_alerts_pages_availability(self):
        """webalert - HTML to text conversion"""

        self.assertEqual("High energy cosmic rays striking atoms at the top of the atmosphere give the rise to showers of particles striking the Earth's surface \nDes rayons cosmiques de haute energie heurtent des atomes dans la haute atmosphere et donnent ainsi naissance a des gerbes de particules projetees sur la surface terrestre  \n 10 May 1999        \nPicture number: CERN-DI-9905005", get_as_text(5))

        self.assertEqual("Quasinormal modes of Reissner-Nordstrom Anti-de Sitter Black Holes  / Wang, B ; Lin, C Y ; Abdalla, E     [hep-th/0003295]   \nComplex frequencies associated with quasinormal modes for large Reissner-Nordstr$\\ddot{o}$m Anti-de Sitter black holes have been computed. [...]   \nFulltext: Fulltext : <PDF>; Fulltext : <PS.GZ> (Fulltext : <additional files>)", get_as_text(74))

TEST_SUITE = make_test_suite(WebAlertWebPagesAvailabilityTest,
                             WebAlertHTMLToTextTest)

if __name__ == "__main__":
    run_test_suite(TEST_SUITE, warn_user=True)
<|endoftext|>


<commit_before>#!/usr/bin/env python3
import html
import json
import requests
import re
import os
from colorama import Fore, Style
from datetime import datetime
from sys import stderr
from xml.etree import ElementTree

def fetch():
    url = 'http://api.openweathermap.org/data/2.5/weather?q=Hsinchui&units=metric&APPID=' + os.environ.get('OPENWEATHER_API_KEY')
    request = requests.get(url)
    return request.text

def fetch_rain():
    url = 'http://opendata.cwb.gov.tw/opendataapi?dataid=O-A0002-001&authorizationkey=' + os.environ.get('CWB_API_KEY')
    request = requests.get(url)
    return request.content

def search_one(pattern, string):
    match = re.search(pattern, string)
    if match is not None:
        match = match.group(1).strip()
    return match

def print_err(text, color=Style.RESET_ALL, bright=False):
    if bright:
        stderr.write(Style.BRIGHT)
    stderr.write(color)
    stderr.write(text)
    stderr.write('\n')

def parse(text, rain_content):
    data = json.loads(text)
    xml_tree = ElementTree.fromstring(rain_content)
    ns = { 'cwb' : 'urn:cwb:gov:tw:cwbcommon:0.1' }
    location = xml_tree.find("./cwb:location/[cwb:stationId='C0D660']", namespaces=ns)
    rain = location.find("./cwb:weatherElement/[cwb:elementName='RAIN']/*/*", namespaces=ns).text
    now = location.find("./cwb:weatherElement/[cwb:elementName='NOW']/*/*", namespaces=ns).text
    date = datetime.fromtimestamp(data['dt']).strftime("%Y/%m/%d %H:%M:%S")
    if not date:
        print_err('ERR: Cannot find observing time. Check service availability.', Fore.RED, bright=True)
        print_err('Original string:')
        print_err(text, Fore.WHITE, bright=True)
        print_err('\n')
        raise Exception('Parse failed')

    return {
        'date': date,
        'temperature': data['main']['temp'],      # (˚C)
        'pressure': data['main']['pressure'],         # (hPa)
        'humidity': data['main']['humidity'],      # (%)
        'wind_speed': data['wind']['speed'],       # (m/s)
        'wind_direction': data['wind']['deg'],   # (˚)
        'rain': 0 if (rain == '-998.00') else float(rain),          # (mm/h)

        'temp_max': data['main']['temp_max'],         # (˚C)
        'temp_min': data['main']['temp_min'],         # (˚C)
        'rain_day': float(now),        # (mm)

        'provider': '中央氣象局',
    }

if __name__ == '__main__':
    from sys import stdout, exit
    text = fetch()
    rain_content = fetch_rain()
    data = parse(text, rain_content)
    json.dump(data, stdout, ensure_ascii=False, indent=2)
<commit_msg>credit<commit_after>#!/usr/bin/env python3
import html
import json
import requests
import re
import os
from colorama import Fore, Style
from datetime import datetime
from sys import stderr
from xml.etree import ElementTree

def fetch():
    url = 'http://api.openweathermap.org/data/2.5/weather?q=Hsinchui&units=metric&APPID=' + os.environ.get('OPENWEATHER_API_KEY')
    request = requests.get(url)
    return request.text

def fetch_rain():
    url = 'http://opendata.cwb.gov.tw/opendataapi?dataid=O-A0002-001&authorizationkey=' + os.environ.get('CWB_API_KEY')
    request = requests.get(url)
    return request.content

def search_one(pattern, string):
    match = re.search(pattern, string)
    if match is not None:
        match = match.group(1).strip()
    return match

def print_err(text, color=Style.RESET_ALL, bright=False):
    if bright:
        stderr.write(Style.BRIGHT)
    stderr.write(color)
    stderr.write(text)
    stderr.write('\n')

def parse(text, rain_content):
    data = json.loads(text)
    xml_tree = ElementTree.fromstring(rain_content)
    ns = { 'cwb' : 'urn:cwb:gov:tw:cwbcommon:0.1' }
    location = xml_tree.find("./cwb:location/[cwb:stationId='C0D660']", namespaces=ns)
    rain = location.find("./cwb:weatherElement/[cwb:elementName='RAIN']/*/*", namespaces=ns).text
    now = location.find("./cwb:weatherElement/[cwb:elementName='NOW']/*/*", namespaces=ns).text
    date = datetime.fromtimestamp(data['dt']).strftime("%Y/%m/%d %H:%M:%S")
    if not date:
        print_err('ERR: Cannot find observing time. Check service availability.', Fore.RED, bright=True)
        print_err('Original string:')
        print_err(text, Fore.WHITE, bright=True)
        print_err('\n')
        raise Exception('Parse failed')

    return {
        'date': date,
        'temperature': data['main']['temp'],      # (˚C)
        'pressure': data['main']['pressure'],         # (hPa)
        'humidity': data['main']['humidity'],      # (%)
        'wind_speed': data['wind']['speed'],       # (m/s)
        'wind_direction': data['wind']['deg'],   # (˚)
        'rain': 0 if (rain == '-998.00') else float(rain),          # (mm/h)

        'temp_max': data['main']['temp_max'],         # (˚C)
        'temp_min': data['main']['temp_min'],         # (˚C)
        'rain_day': float(now),        # (mm)

        'provider': '中央氣象局, OpenWeatherMap under CC BY-SA 4.0',
    }

if __name__ == '__main__':
    from sys import stdout, exit
    text = fetch()
    rain_content = fetch_rain()
    data = parse(text, rain_content)
    json.dump(data, stdout, ensure_ascii=False, indent=2)
<|endoftext|>


<commit_before>from unittest import TestCase

from mongotools import mim
from mongotools.semaphore import Semaphore

class SemaphoreTestCase(TestCase):

    def setUp(self):
        self.bind = mim.Connection.get()
        self.bind.drop_all()
        self.semaphore = Semaphore(self.bind.db, 1, 'counter', 1)

    def test_acquire(self):
        self.assertTrue(self.semaphore.acquire(), msg='Failed to acquire intial semaphore')
        self.assertEqual(self.semaphore._counter, 0)
        self.assertFalse(self.semaphore.acquire(), msg='Failed to block after first acquire')

    def test_release(self):
        self.semaphore.acquire()
        self.semaphore.release()
        self.assertEqual(self.semaphore._counter, 1)

class BasicSemaphoreTestCase(TestCase):

    def setUp(self):
        self.bind = mim.Connection.get()
        self.bind.drop_all()
        self.semaphore = Semaphore(self.bind.db, 1, 'counter', 1)

    def test_semaphore_name(self):
        pass
<commit_msg>Updated test cases<commit_after>from unittest import TestCase

from mongotools import mim
from mongotools.semaphore import Semaphore

class SemaphoreBinaryTestCase(TestCase):

    def setUp(self):
        self.bind = mim.Connection.get()
        self.bind.drop_all()
        self.semaphore = Semaphore(self.bind.db, 1, 'counter', 1)

    def test_acquire(self):
        self.assertTrue(self.semaphore.acquire(), msg='Failed to acquire intial semaphore')

    def test_release(self):
        self.semaphore.acquire()
        self.semaphore.release()

    def test_sequence(self):
        self.assertTrue(self.semaphore.acquire(), msg='Failed to acquire intial semaphore')
        self.assertFalse(self.semaphore.acquire(), msg='Failed to block on second acquire')
        self.semaphore.release()
        self.assertTrue(self.semaphore.acquire(), msg='Failed to reacquire after release')
        for i in xrange(10):
            self.semaphore.release()
            self.assertTrue(self.semaphore.acquire())
            self.semaphore.release()

class SemaphoreBinaryTestCase(TestCase):

    _MAX = 5

    def setUp(self):
        self.bind = mim.Connection.get()
        self.bind.drop_all()
        self.semaphore = Semaphore(self.bind.db, 1, 'counter', self._MAX)

    def test_acquire(self):
        for i in xrange(self._MAX):
            self.assertTrue(self.semaphore.acquire(), msg='Failed to acquire intial semaphore')
        self.assertFalse(self.semaphore.acquire())

class BasicSemaphoreTestCase(TestCase):

    def setUp(self):
        self.bind = mim.Connection.get()
        self.bind.drop_all()
        self.semaphore = Semaphore(self.bind.db, 1, 'counter', 1)

    def test_counter_name(self):
        self.assertEquals(self.semaphore._counter, 'counter')
<|endoftext|>


<commit_before>from kvlist import KVList
from binary import getbyte

def skip_space(s):
    while True:
        c = s.read(1)
        if not c.isspace():
            break
    return c

def getstring(s):
    lc = []
    while True:
        c = getbyte(s)
        if c == "\"":
            break
        lc.append(c)
    return "".join(lc)

def load(s):
    items = KVList()
    while True:
        c = skip_space(s)
        if c == "":
            break
        elif c == "\"":
            k, v = parse_item(s, [])
            items[k] = v
        else:
            assert False, u"Unexpected character '{}'".format(c)
    return items

def parse_dict(s, context):
    d = KVList()
    while True:
        c = skip_space(s)
        if c == '}':
            break
        elif c == '"':
            k, v = parse_item(s, context)
            d[k] = v
        else:
            assert False, u"Expected '\"' or '}', got '{}' in {}".format(c, context)
    return d

def parse_item(s, context):
    k = getstring(s)
    c = skip_space(s)
    if c == '"':
        v = getstring(s)
    elif c == '{':
        v = parse_dict(s, context+[k])
    else:
        assert False, u"Expected a string or a dict, got '{}' in {}".format(c, repr(context))
    return k, v

def indent(i, s):
    for j in xrange(i):
        s.write('\t')

def dump(d, s, i=0):
    for k, v in d:
        indent(i, s)
        s.write('"')
        s.write(k)
        s.write('"')
        if isinstance(v, KVList):
            s.write('\n')
            indent(i, s)
            s.write('{')
            s.write('\n')
            dump(v, s, i+1)
            indent(i, s)
            s.write('}')
            s.write('\n')
        elif isinstance(v, basestring):
            s.write('\t\t')
            s.write('"')
            s.write(v)
            s.write('"')
            s.write('\n')
        else:
            assert False, u"Expected KVList or basestring, got {}".format(type(v))

if __name__ == "__main__":
    with open("items_game.txt", "rb") as input:
        d = load(input)
    with open("items_game_dump.txt", "wb") as output:
        dump(d, output)<commit_msg>vdf comments<commit_after>from kvlist import KVList
from binary import getbyte

def skip_space(s):
    while True:
        c = s.read(1)
        if c == "/":
            while True:
                c = s.read(1)
                if c == "\n":
                    break
        elif not c.isspace():
            break
    return c

def getstring(s):
    lc = []
    while True:
        c = getbyte(s)
        if c == "\"":
            break
        lc.append(c)
    return "".join(lc)

def load(s):
    items = KVList()
    while True:
        c = skip_space(s)
        if c == "":
            break
        elif c == "\"":
            k, v = parse_item(s, [])
            items[k] = v
        else:
            assert False, u"Unexpected character '{}'".format(c)
    return items

def parse_dict(s, context):
    d = KVList()
    while True:
        c = skip_space(s)
        if c == '}':
            break
        elif c == '"':
            k, v = parse_item(s, context)
            d[k] = v
        else:
            assert False, u"Expected '\"' or '}', got '{}' in {}".format(c, context)
    return d

def parse_item(s, context):
    k = getstring(s)
    c = skip_space(s)
    if c == '"':
        v = getstring(s)
    elif c == '{':
        v = parse_dict(s, context+[k])
    else:
        assert False, u"Expected a string or a dict, got '{}' in {}".format(c, repr(context))
    return k, v

def indent(i, s):
    for j in xrange(i):
        s.write('\t')

def dump(d, s, i=0):
    for k, v in d:
        indent(i, s)
        s.write('"')
        s.write(k)
        s.write('"')
        if isinstance(v, KVList):
            s.write('\n')
            indent(i, s)
            s.write('{')
            s.write('\n')
            dump(v, s, i+1)
            indent(i, s)
            s.write('}')
            s.write('\n')
        elif isinstance(v, basestring):
            s.write('\t\t')
            s.write('"')
            s.write(v)
            s.write('"')
            s.write('\n')
        else:
            assert False, u"Expected KVList or basestring, got {}".format(type(v))

if __name__ == "__main__":
    with open("items_game.txt", "rb") as input:
        d = load(input)
    with open("items_game_dump.txt", "wb") as output:
        dump(d, output)<|endoftext|>


<commit_before>#!/usr/local/bin/python3

import requests
from datetime import date, timedelta
from multiprocessing.dummy import Pool

with open('metoffice_api_key') as keyfile:
    api_key = keyfile.read()

weather_types = {'NA': 'Not available',
                 0: 'Clear night',
                 1: 'Sunny day',
                 2: 'Partly cloudy',        # night
                 3: 'Partly cloudy',        # day
                 4: 'Not used',
                 5: 'Mist',
                 6: 'Fog',
                 7: 'Cloudy',
                 8: 'Overcast',
                 9: 'Light rain shower',    # night
                 10: 'Light rain shower',   # day
                 11: 'Drizzle',
                 12: 'Light rain',
                 13: 'Heavy rain shower',   # night
                 14: 'Heavy rain shower',   # day
                 15: 'Heavy rain',
                 16: 'Sleet shower',        # night
                 17: 'Sleet shower',        # day
                 18: 'Sleet',
                 19: 'Hail shower',         # night
                 20: 'Hail shower',         # day
                 21: 'Hail',
                 22: 'Light snow shower',   # night
                 23: 'Light snow shower',   # day
                 24: 'Light snow',
                 25: 'Heavy snow shower',   # night
                 26: 'Heavy snow shower',   # day
                 27: 'Heavy snow',
                 28: 'Thunder shower',      # night
                 29: 'Thunder shower',      # day
                 30: 'Thunder'}

forecast_codes = {
    'FDm': {'name': 'Feels like',
            'units': '°C'},
    'Dm': {'name': 'Max temp',
           'units': '°C'},
    'Gn': {'name': 'Wind gusts',
           'units': 'mph'},
    'Hn': {'name': 'Humidity',
           'units': '%'},
    'V': {'name': 'Visibility',
          'units': ''},
    'D': {'name': 'Wind direction',
          'units': ''},
    'S': {'name': 'Wind speed',
          'units': 'mph'},
    'U': {'name': 'UV',
          'units': ''},
    'W': {'name': 'Weather type',
          'units': ''},
    'PPd': {'name': 'Precipitation probability',
            'units': '%'},
}

urls = {
    'base': 'http://datapoint.metoffice.gov.uk/public/data/',
    'forecast': 'val/wxfcs/all/json/{location}',
    'summary': 'txt/wxfcs/regionalforecast/json/{location}',
    }

location_tuples = (
    ('Aberdeen', 310170),
    ('Birmingham', 310002),
    ('Cardiff', 350758),
    ('Edinburgh', 351351),
    ('Glasgow', 310009),
    ('Liverpool', 310012),
    ('London', 352409),
    ('Manchester', 310013),
    ('Newcastle', 352793),
    ('Norwich', 310115),
    ('Plymouth', 310016),
    ('Sheffield', 353467),
    ('Southampton', 353595),
    )

location_dicts = [{'name': a, 'code': b} for a, b in location_tuples]

def fetch_forecast(location_code, target_date):
    url = urls['base'] + urls['forecast'].format(location=location_code)
    params = {'key': api_key,
              'res': 'daily',
              'time': target_date}
    response = requests.request('GET', url, params=params)
    conditions = response.json()['SiteRep']['DV']['Location']['Period']['Rep']
    day_forecast = next(f for f in conditions if f['$'] == 'Day')
    return day_forecast

def parse_forecast(forecast):
    working_dict = {}
    for k, v in forecast.items():
        if k == '$':
            continue
        name = forecast_codes[k]['name']
        units = forecast_codes[k]['units']
        if name == 'Weather type':
            working_dict[name] = weather_types[int(v)]
        else:
            working_dict[name] = v + units
    return working_dict

def build_weather_string(parsed_dict):
    weather_template = '''\
{Weather type}
{temp}
Wind {Wind speed} {Wind direction}{precip}'''
    temp_string = parsed_dict['Max temp'] + ' max'
    if parsed_dict['Feels like'] != parsed_dict['Max temp']:
        temp_string += ', feels like {}'.format(parsed_dict['Feels like'])

    precip_chance = parsed_dict['Precipitation probability']
    if int(precip_chance.replace('%', '')) < 20:
        precip_string = ''
    else:
        precip_string = '\n{} chance of rain'.format(precip_chance)

    weather_string = weather_template.format(temp=temp_string,
                                             precip=precip_string,
                                             **parsed_dict)
    return weather_string

def add_daily_forecast_to_dict(location, forecast_date):
    raw_forecast = fetch_forecast(location['code'], forecast_date)
    parsed = parse_forecast(raw_forecast)
    location[forecast_date] = build_weather_string(parsed)

def date_string(d):
    return d.isoformat() + 'T00:00:00Z'

if __name__ == '__main__':
    today = date.today()
    tomorrow = date_string(today + timedelta(1))
    if today.weekday() == 4:    # today is Friday
        next_day = date_string(today + timedelta(2))
    else:
        next_day = None

    for loc in location_dicts:
        parsed = parse_forecast(fetch_forecast(loc['code'], tomorrow))
        location[tomorrow] = build_weather_string(parsed)

    # with Pool() as pool:
    #     pool.starmap(add_daily_forecast_to_dict,
    #                  [(loc, tomorrow) for loc in location_dicts])
<commit_msg>weather.py: Add 3-5 day outlook function<commit_after>#!/usr/local/bin/python3

import requests
from datetime import date, timedelta
from multiprocessing.dummy import Pool

with open('metoffice_api_key') as keyfile:
    api_key = keyfile.read()

weather_types = {'NA': 'Not available',
                 0: 'Clear night',
                 1: 'Sunny day',
                 2: 'Partly cloudy',        # night
                 3: 'Partly cloudy',        # day
                 4: 'Not used',
                 5: 'Mist',
                 6: 'Fog',
                 7: 'Cloudy',
                 8: 'Overcast',
                 9: 'Light rain shower',    # night
                 10: 'Light rain shower',   # day
                 11: 'Drizzle',
                 12: 'Light rain',
                 13: 'Heavy rain shower',   # night
                 14: 'Heavy rain shower',   # day
                 15: 'Heavy rain',
                 16: 'Sleet shower',        # night
                 17: 'Sleet shower',        # day
                 18: 'Sleet',
                 19: 'Hail shower',         # night
                 20: 'Hail shower',         # day
                 21: 'Hail',
                 22: 'Light snow shower',   # night
                 23: 'Light snow shower',   # day
                 24: 'Light snow',
                 25: 'Heavy snow shower',   # night
                 26: 'Heavy snow shower',   # day
                 27: 'Heavy snow',
                 28: 'Thunder shower',      # night
                 29: 'Thunder shower',      # day
                 30: 'Thunder'}

forecast_codes = {
    'FDm': {'name': 'Feels like',
            'units': '°C'},
    'Dm': {'name': 'Max temp',
           'units': '°C'},
    'Gn': {'name': 'Wind gusts',
           'units': 'mph'},
    'Hn': {'name': 'Humidity',
           'units': '%'},
    'V': {'name': 'Visibility',
          'units': ''},
    'D': {'name': 'Wind direction',
          'units': ''},
    'S': {'name': 'Wind speed',
          'units': 'mph'},
    'U': {'name': 'UV',
          'units': ''},
    'W': {'name': 'Weather type',
          'units': ''},
    'PPd': {'name': 'Precipitation probability',
            'units': '%'},
}

urls = {
    'base': 'http://datapoint.metoffice.gov.uk/public/data/',
    'forecast': 'val/wxfcs/all/json/{location}',
    'summary': 'txt/wxfcs/regionalforecast/json/{location}',
    }

location_tuples = (
    ('Aberdeen', 310170),
    ('Birmingham', 310002),
    ('Cardiff', 350758),
    ('Edinburgh', 351351),
    ('Glasgow', 310009),
    ('Liverpool', 310012),
    ('London', 352409),
    ('Manchester', 310013),
    ('Newcastle', 352793),
    ('Norwich', 310115),
    ('Plymouth', 310016),
    ('Sheffield', 353467),
    ('Southampton', 353595),
    )

location_dicts = [{'name': a, 'code': b} for a, b in location_tuples]

def fetch_forecast(location_code, target_date):
    url = urls['base'] + urls['forecast'].format(location=location_code)
    params = {'key': api_key,
              'res': 'daily',
              'time': target_date}
    response = requests.request('GET', url, params=params)
    conditions = response.json()['SiteRep']['DV']['Location']['Period']['Rep']
    day_forecast = next(f for f in conditions if f['$'] == 'Day')
    return day_forecast

def parse_forecast(forecast):
    working_dict = {}
    for k, v in forecast.items():
        if k == '$':
            continue
        name = forecast_codes[k]['name']
        units = forecast_codes[k]['units']
        if name == 'Weather type':
            working_dict[name] = weather_types[int(v)]
        else:
            working_dict[name] = v + units
    return working_dict

def build_weather_string(parsed_dict):
    weather_template = '''\
{Weather type}
{temp}
Wind {Wind speed} {Wind direction}{precip}'''
    temp_string = parsed_dict['Max temp'] + ' max'
    if parsed_dict['Feels like'] != parsed_dict['Max temp']:
        temp_string += ', feels like {}'.format(parsed_dict['Feels like'])

    precip_chance = parsed_dict['Precipitation probability']
    if int(precip_chance.replace('%', '')) < 20:
        precip_string = ''
    else:
        precip_string = '\n{} chance of rain'.format(precip_chance)

    weather_string = weather_template.format(temp=temp_string,
                                             precip=precip_string,
                                             **parsed_dict)
    return weather_string

def add_daily_forecast_to_dict(location, forecast_date):
    raw_forecast = fetch_forecast(location['code'], forecast_date)
    parsed = parse_forecast(raw_forecast)
    location[forecast_date] = build_weather_string(parsed)

def date_string(d):
    return d.isoformat() + 'T00:00:00Z'

def fetch_uk_outlook():
    url = urls['base'] + urls['summary'].format(location=515)
    params = {'key': api_key}
    response = requests.request('GET', url, params=params)
    periods = response.json()['RegionalFcst']['FcstPeriods']['Period']
    summary_dict = next(p for p in periods if p['id'] == 'day3to5')
    outlook = summary_dict['Paragraph']['$']
    return outlook

if __name__ == '__main__':
    outlook_text = fetch_uk_outlook()

    today = date.today()
    tomorrow = date_string(today + timedelta(1))
    if today.weekday() == 4:    # today is Friday
        next_day = date_string(today + timedelta(2))
    else:
        next_day = None

    for loc in location_dicts:
        parsed = parse_forecast(fetch_forecast(loc['code'], tomorrow))
        location[tomorrow] = build_weather_string(parsed)

    # with Pool() as pool:
    #     pool.starmap(add_daily_forecast_to_dict,
    #                  [(loc, tomorrow) for loc in location_dicts])

<|endoftext|>


<commit_before># Copyright 2010 Google Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""Tests for bubble.py."""

from google.appengine.api import users

from bubble import HospitalValueInfoExtractor, ValueInfoExtractor
from utils import db, HIDDEN_ATTRIBUTE_NAMES

import django.utils.translation

import bubble
import datetime
import logging
import model
import os
import unittest
import utils

SAN_FRANCISCO = {'lat': 40.7142, 'lon': -74.0064}

def fake_get_message(ns, n):
    message = model.Message(namespace = ns, name = n)
    if ns == 'attribute_value' and n == 'fake_to_localize':
        message.en = 'fake_localized'
    else:
        message.en = 'foo'
    django_locale = 'en'

    return message and getattr(message, django_locale) or n

class BubbleTest(unittest.TestCase):
    def setUp(self):
        self.real_auth_domain = os.environ.get('AUTH_DOMAIN')
        os.environ['AUTH_DOMAIN'] = 'test'
        self.real_get_message = bubble.get_message
        bubble.get_message = fake_get_message

    def tearDown(self):
        bubble.get_message = self.real_get_message
        if self.real_auth_domain:
            os.environ['AUTH_DOMAIN'] = self.real_auth_domain
        else:
            os.environ['AUTH_DOMAIN'] = ''

    def test_format(self):
        time = datetime.datetime(2010, 6, 2, 13, 21, 13, 97435)
        pt = utils.db.GeoPt(SAN_FRANCISCO['lat'], SAN_FRANCISCO['lon'])
        assert bubble.format(u'123\u26CC') == '123\xe2\x9b\x8c'
        assert bubble.format('12\n3') == bubble.format('12 3')
        assert bubble.format('123') == '123'
        assert bubble.format(['1', '2', '3']) == bubble.format('1, 2, 3') \
            == '1, 2, 3'
        assert bubble.format(time) == '2010-06-02 08:21:13 -05:00'
        assert bubble.format(pt) == '40.7142\xc2\xb0 N, 74.0064\xc2\xb0 W'
        assert bubble.format(True) == bubble.format(_('Yes')) == _('Yes')
        assert bubble.format(False) == bubble.format(_('No')) == _('No')
        assert bubble.format({'hey!' : 1}) == {'hey!' : 1}
        assert bubble.format(None) == u'\u2013'.encode('utf-8')
        assert bubble.format('') == u'\u2013'.encode('utf-8')
        assert bubble.format([]) == u'\u2013'.encode('utf-8')
        assert bubble.format({}) == u'\u2013'.encode('utf-8')
        assert bubble.format(13) == 13
        assert bubble.format(0) == 0
        assert bubble.format('fake_to_localize', True) == 'fake_localized'
        assert bubble.format(['fake_to_localize'], True == ['fake_localized'])

    def test_value_info_extractor(self):
        f = model.Facility(key_name='example.org/123', type='hospital')
        f.set_attribute('title', 'title_foo', datetime.datetime.now(),
                        users.User('test@example.com'),
                        'nickname_foo', 'affiliation_foo', 'comment_foo')
        f.set_attribute('attribute_value', 'fake_to_localize',
                        datetime.datetime.now(),
                        users.User('test@example.com'),
                        'nickname_foo', 'affiliation_foo', 'comment_foo')

        vai = ValueInfoExtractor(['title'], ['attribute_value'])
        (special, general, details) = vai.extract(f, ['title'])

        assert special['title'].raw == 'title_foo'
        assert general == []
        assert details[0].raw == 'title_foo'

        (special, general, details) = vai.extract(f, ['attribute_value'])

        assert general[0].raw == 'fake_to_localize'
        assert general[0].value == 'fake_localized'
        assert general[0].label == 'foo'

    def test_hospital_value_info_extractor(self):
        user = users.User('test@example.com')
        now = datetime.datetime(2010, 6, 11, 14, 26, 52, 906773)
        nickname = 'nickname_foo'
        affiliation = 'affiliation_foo'
        comment = 'comment_foo'

        f = model.Facility(key_name='example.org/123', type='hospital')
        f.set_attribute('title', 'title_foo', now, user, nickname, affiliation,
                        comment)
        f.set_attribute(HIDDEN_ATTRIBUTE_NAMES[0], 'hidden_value_foo', now,
                        user, nickname, affiliation, comment)
        f.set_attribute('organization_name', 'value_foo', now, user, nickname,
                        affiliation, comment)

        attrs = ['title', 'organization_name', HIDDEN_ATTRIBUTE_NAMES[0]]
        vai = HospitalValueInfoExtractor()
        (special, general, details) = vai.extract(f, attrs)

        assert special['title'].date == '2010-06-11 09:26:52 -05:00'
        assert special['title'].raw == 'title_foo'
        assert HIDDEN_ATTRIBUTE_NAMES[0] not in special
        assert sorted(special) == sorted(vai.special_attribute_names)
        assert len(general) == 1
        assert len(details) == 2
        assert general[0].value == 'value_foo'
        for detail in details:
            assert detail.value == 'title_foo' or detail.value == 'value_foo'
            assert detail.value != 'hidden_value_foo'

'''    def test_vai_get_value_info(self):
        f = model.Facility(key_name='example.org/123', type='hospital')
        f.set_attribute('title', 'title_foo', datetime.datetime(2010, 06, 01),
                        users.User('test@example.com'),
                        'nickname_foo', 'affiliation_foo\n', 'comment_\nfoo')
        vai = ValueInfoExtractor(['title'], [])
        vi = vai.get_value_info(f, 'title')

        assert vi.label == 'title'
        assert vi.raw == 'title_foo'
        assert vi.localize == u'\u2013'
        assert vi.author == 'test@example.com'
        assert vi.affiliation == 'affiliation_foo '
        assert vi.comment == 'comment_ foo'
        assert vi.date == '2010-05-31 19:00:00 -05:00'
'''
<commit_msg>remove bad code remnant<commit_after># Copyright 2010 Google Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""Tests for bubble.py."""

from google.appengine.api import users

from bubble import HospitalValueInfoExtractor, ValueInfoExtractor
from utils import db, HIDDEN_ATTRIBUTE_NAMES

import django.utils.translation

import bubble
import datetime
import logging
import model
import os
import unittest
import utils

SAN_FRANCISCO = {'lat': 40.7142, 'lon': -74.0064}

def fake_get_message(ns, n):
    message = model.Message(namespace = ns, name = n)
    if ns == 'attribute_value' and n == 'fake_to_localize':
        message.en = 'fake_localized'
    else:
        message.en = 'foo'
    django_locale = 'en'

    return message and getattr(message, django_locale) or n

class BubbleTest(unittest.TestCase):
    def setUp(self):
        self.real_auth_domain = os.environ.get('AUTH_DOMAIN')
        os.environ['AUTH_DOMAIN'] = 'test'
        self.real_get_message = bubble.get_message
        bubble.get_message = fake_get_message

    def tearDown(self):
        bubble.get_message = self.real_get_message
        if self.real_auth_domain:
            os.environ['AUTH_DOMAIN'] = self.real_auth_domain
        else:
            os.environ['AUTH_DOMAIN'] = ''

    def test_format(self):
        time = datetime.datetime(2010, 6, 2, 13, 21, 13, 97435)
        pt = utils.db.GeoPt(SAN_FRANCISCO['lat'], SAN_FRANCISCO['lon'])
        assert bubble.format(u'123\u26CC') == '123\xe2\x9b\x8c'
        assert bubble.format('12\n3') == bubble.format('12 3')
        assert bubble.format('123') == '123'
        assert bubble.format(['1', '2', '3']) == bubble.format('1, 2, 3') \
            == '1, 2, 3'
        assert bubble.format(time) == '2010-06-02 08:21:13 -05:00'
        assert bubble.format(pt) == '40.7142\xc2\xb0 N, 74.0064\xc2\xb0 W'
        assert bubble.format(True) == bubble.format(_('Yes')) == _('Yes')
        assert bubble.format(False) == bubble.format(_('No')) == _('No')
        assert bubble.format({'hey!' : 1}) == {'hey!' : 1}
        assert bubble.format(None) == u'\u2013'.encode('utf-8')
        assert bubble.format('') == u'\u2013'.encode('utf-8')
        assert bubble.format([]) == u'\u2013'.encode('utf-8')
        assert bubble.format({}) == u'\u2013'.encode('utf-8')
        assert bubble.format(13) == 13
        assert bubble.format(0) == 0
        assert bubble.format('fake_to_localize', True) == 'fake_localized'
        assert bubble.format(['fake_to_localize'], True == ['fake_localized'])

    def test_value_info_extractor(self):
        f = model.Facility(key_name='example.org/123', type='hospital')
        f.set_attribute('title', 'title_foo', datetime.datetime.now(),
                        users.User('test@example.com'),
                        'nickname_foo', 'affiliation_foo', 'comment_foo')
        f.set_attribute('attribute_value', 'fake_to_localize',
                        datetime.datetime.now(),
                        users.User('test@example.com'),
                        'nickname_foo', 'affiliation_foo', 'comment_foo')

        vai = ValueInfoExtractor(['title'], ['attribute_value'])
        (special, general, details) = vai.extract(f, ['title'])

        assert special['title'].raw == 'title_foo'
        assert general == []
        assert details[0].raw == 'title_foo'

        (special, general, details) = vai.extract(f, ['attribute_value'])

        assert general[0].raw == 'fake_to_localize'
        assert general[0].value == 'fake_localized'
        assert general[0].label == 'foo'

    def test_hospital_value_info_extractor(self):
        user = users.User('test@example.com')
        now = datetime.datetime(2010, 6, 11, 14, 26, 52, 906773)
        nickname = 'nickname_foo'
        affiliation = 'affiliation_foo'
        comment = 'comment_foo'

        f = model.Facility(key_name='example.org/123', type='hospital')
        f.set_attribute('title', 'title_foo', now, user, nickname, affiliation,
                        comment)
        f.set_attribute(HIDDEN_ATTRIBUTE_NAMES[0], 'hidden_value_foo', now,
                        user, nickname, affiliation, comment)
        f.set_attribute('organization_name', 'value_foo', now, user, nickname,
                        affiliation, comment)

        attrs = ['title', 'organization_name', HIDDEN_ATTRIBUTE_NAMES[0]]
        vai = HospitalValueInfoExtractor()
        (special, general, details) = vai.extract(f, attrs)

        assert special['title'].date == '2010-06-11 09:26:52 -05:00'
        assert special['title'].raw == 'title_foo'
        assert HIDDEN_ATTRIBUTE_NAMES[0] not in special
        assert sorted(special) == sorted(vai.special_attribute_names)
        assert len(general) == 1
        assert len(details) == 2
        assert general[0].value == 'value_foo'
        for detail in details:
            assert detail.value == 'title_foo' or detail.value == 'value_foo'
            assert detail.value != 'hidden_value_foo'
<|endoftext|>


<commit_before>import json
from skytap.models.SkytapResource import SkytapResource

class UserData(SkytapResource):
    def __init__(self, contents):
        super(UserData, self).__init__(contents)

    def __str__(self):
        return self.contents

    def _clean_value(self, key, value):
        default_time = 3
        default_delay_min = 0
        default_delay_max = 31

        # Check value for shutdown_time and decide what to do with it
        if key == "shutdown_time":
            try:
                if value == "-":
                    # Permanent exclusion case
                    pass
                elif (int(value) < 0 or int(value) > 23):
                    # Should be valid time
                    value = str(default_time)
            except ValueError:
                # Not an int, revert to default
                value = str(default_time)

        # Check value for shutdown_delay and decide what to do with it
        if key == "shutdown_delay":
            try:
                if int(value) < default_delay_min:
                    # Delay smaller than minimum, set to 0
                    value = str(default_delay_min)
                elif int(value) > default_delay_max:
                    # Delay larger than maximum, set to 31
                    value = str(default_delay_max)
            except ValueError:
                # Delay is not a number, change to 0
                value = str(default_delay_min)

        # If this is valid, leave it alone. Otherwise, change the value to "-",
        # which triggers the variable itself to be removed.
        if key == "delete_environment":
            try:
                int(value)
            except ValueError:
                value = "error"

        return value

    def _get_values(self, contents):
        lines = contents.split("\n")

        values = {}

        for i in lines:
            tokens = i.split()

            if len(tokens) < 2:
                continue

            # Check for valid YAML formatting in first and second tokens in each
            # line, then add those values to dict.
            if (tokens[0].endswith(":") and "#" not in tokens[0] and
                    len(tokens) > 1 and "#" not in tokens[1]):
                    values[tokens[0][:-1]] = self._clean_value(tokens[0][:-1],
                                                               tokens[1])
                    self.data[tokens[0][:-1]] = values[tokens[0][:-1]]

        return values

    def _calculate_custom_data(self):
        if self.contents:
            values = self._get_values(self.contents)
        else:
            self.data["contents"] = ""
<commit_msg>UserData variables are now made integers if in number form<commit_after>import json
from skytap.models.SkytapResource import SkytapResource

class UserData(SkytapResource):
    def __init__(self, contents):
        super(UserData, self).__init__(contents)

    def __str__(self):
        return self.contents

    def _clean_value(self, key, value):
        default_time = 3
        default_delay_min = 0
        default_delay_max = 31

        # Check value for shutdown_time and decide what to do with it
        if key == "shutdown_time":
            try:
                if value == "-":
                    # Permanent exclusion case
                    pass
                elif (int(value) < 0 or int(value) > 23):
                    # Should be valid time
                    value = str(default_time)
            except ValueError:
                # Not an int, revert to default
                value = str(default_time)

        # Check value for shutdown_delay and decide what to do with it
        if key == "shutdown_delay":
            try:
                if int(value) < default_delay_min:
                    # Delay smaller than minimum, set to 0
                    value = str(default_delay_min)
                elif int(value) > default_delay_max:
                    # Delay larger than maximum, set to 31
                    value = str(default_delay_max)
            except ValueError:
                # Delay is not a number, change to 0
                value = str(default_delay_min)

        # If this is valid, leave it alone. Otherwise, change the value to "-",
        # which triggers the variable itself to be removed.
        if key == "delete_environment":
            try:
                int(value)
            except ValueError:
                value = "error"

        return value

    def _get_values(self, contents):
        lines = contents.split("\n")

        values = {}

        for i in lines:
            tokens = i.split()

            if len(tokens) < 2:
                continue

            # Check for valid YAML formatting in first and second tokens in each
            # line, then add those values to dict.
            if (tokens[0].endswith(":") and "#" not in tokens[0] and
                    len(tokens) > 1 and "#" not in tokens[1]):
                    values[tokens[0][:-1]] = self._clean_value(tokens[0][:-1],
                                                               tokens[1])
                    # If variable is a number, make it integer
                    try:
                        values[tokens[0][:-1]] = int(values[tokens[0][:-1]])
                    except ValueError:
                        pass

                    self.data[tokens[0][:-1]] = values[tokens[0][:-1]]

        return values

    def _calculate_custom_data(self):
        if self.contents:
            values = self._get_values(self.contents)
        else:
            self.data["contents"] = ""
<|endoftext|>


<commit_before>from flask import Flask, Module, Response
from flask import abort, json, redirect, request, url_for, g

from .model import Annotation
from authz import authorize, ACTION
from . import auth

__all__ = ["store"]

store = Module(__name__)

from flask import current_app

# We define our own jsonify rather than using flask.jsonify because we wish
# to jsonify arbitrary objects (e.g. index returns a list) rather than kwargs.
def jsonify(obj, *args, **kwargs):
    res = json.dumps(obj, indent=None if request.is_xhr else 2)
    return Response(res, mimetype='application/json', *args, **kwargs)

def unjsonify(str):
    return json.loads(str)

def get_current_userid():
    return auth.get_request_userid(request)

@store.before_request
def before_request():
    g.account_id = request.headers.get('x-annotator-account-id', '')
    if current_app.config['AUTH_ON'] and not request.method == 'GET' and not auth.verify_request(request):
        return jsonify("Cannot authorise request. Perhaps you didn't send the x-annotator headers?", status=401)

@store.after_request
def after_request(response):
    response.headers['Access-Control-Allow-Origin']   = '*'
    # response.headers['Access-Control-Allow-Headers'] = '*'
    response.headers['Access-Control-Allow-Headers'] = 'X-Requested-With, Content-Type, X-Annotator-Account-Id, X-Annotator-User-Id, X-Annotator-Auth-Token-Valid-Until, X-Annotator-Auth-Token'
    response.headers['Access-Control-Expose-Headers'] = 'Location'
    response.headers['Access-Control-Allow-Methods']  = 'GET, POST, PUT, DELETE'
    response.headers['Access-Control-Max-Age']        = '86400'

    return response

@store.route('/')
def home():
    return jsonify('Annotator Store API')

# INDEX
@store.route('/annotations')
def index():
    annotations = [row.doc for row in Annotation.search() if authorize(a, 'read', get_current_userid())]
    return jsonify(annotations)

# CREATE
@store.route('/annotations', methods=['POST'])
def create_annotation():
    if request.json:
        annotation = Annotation.from_dict(request.json)
        annotation.account_id = g.account_id
        annotation.save()
        return jsonify(annotation.to_dict())
    else:
        return jsonify('No parameters given. Annotation not created.', status=400)

# READ
@store.route('/annotations/<id>')
def read_annotation(id):
    annotation = Annotation.get(id)

    if not annotation:
        return jsonify('Annotation not found.', status=404)
    elif authorize(annotation, 'read', get_current_userid()):
        return jsonify(annotation.to_dict())
    else:
        return jsonify('Could not authorise request. Read not allowed', status=401)

# UPDATE
@store.route('/annotations/<id>', methods=['PUT'])
def update_annotation(id):
    annotation = Annotation.get(id)

    if not annotation:
        return jsonify('Annotation not found. No update performed.', status=404)

    elif request.json and authorize(annotation, 'update', get_current_userid()):
        updated = Annotation.from_dict(request.json)
        if updated.permissions != annotation.permissions:
            if not authorize(annotation, ACTION.ADMIN, get_current_userid()):
                return jsonify('Could not authorise request (permissions change). No update performed', status=401)
        updated.save()
        return jsonify(updated.to_dict())
    else:
        return jsonify('Could not authorise request. No update performed', status=401)

# DELETE
@store.route('/annotations/<id>', methods=['DELETE'])
def delete_annotation(id):
    annotation = Annotation.get(id)

    if not annotation:
        return jsonify('Annotation not found. No delete performed.', status=404)

    elif authorize(annotation, 'delete', get_current_userid()):
        annotation.delete()
        return None, 204

    else:
        return jsonify('Could not authorise request. No update performed', status=401)

# Search
@store.route('/search')
def search_annotations():
    kwargs = dict(request.args.items())
    # TODO: limit results returned to max 200
    results = [ x.to_dict() for x in Annotation.search(**kwargs) ]
    # TODO: a proper count(*) for this query
    total = len(results)
    qrows = {
        'total': total,
        'rows': results,
    }
    return jsonify(qrows)

<commit_msg>[store][xs]: allow setting of account_id from annotation if not in headers.<commit_after>from flask import Flask, Module, Response
from flask import abort, json, redirect, request, url_for, g

from .model import Annotation
from authz import authorize, ACTION
from . import auth

__all__ = ["store"]

store = Module(__name__)

from flask import current_app

# We define our own jsonify rather than using flask.jsonify because we wish
# to jsonify arbitrary objects (e.g. index returns a list) rather than kwargs.
def jsonify(obj, *args, **kwargs):
    res = json.dumps(obj, indent=None if request.is_xhr else 2)
    return Response(res, mimetype='application/json', *args, **kwargs)

def unjsonify(str):
    return json.loads(str)

def get_current_userid():
    return auth.get_request_userid(request)

@store.before_request
def before_request():
    g.account_id = request.headers.get('x-annotator-account-id', '')
    if current_app.config['AUTH_ON'] and not request.method == 'GET' and not auth.verify_request(request):
        return jsonify("Cannot authorise request. Perhaps you didn't send the x-annotator headers?", status=401)

@store.after_request
def after_request(response):
    response.headers['Access-Control-Allow-Origin']   = '*'
    # response.headers['Access-Control-Allow-Headers'] = '*'
    response.headers['Access-Control-Allow-Headers'] = 'X-Requested-With, Content-Type, X-Annotator-Account-Id, X-Annotator-User-Id, X-Annotator-Auth-Token-Valid-Until, X-Annotator-Auth-Token'
    response.headers['Access-Control-Expose-Headers'] = 'Location'
    response.headers['Access-Control-Allow-Methods']  = 'GET, POST, PUT, DELETE'
    response.headers['Access-Control-Max-Age']        = '86400'

    return response

@store.route('/')
def home():
    return jsonify('Annotator Store API')

# INDEX
@store.route('/annotations')
def index():
    annotations = [row.doc for row in Annotation.search() if authorize(a, 'read', get_current_userid())]
    return jsonify(annotations)

# CREATE
@store.route('/annotations', methods=['POST'])
def create_annotation():
    if request.json:
        annotation = Annotation.from_dict(request.json)
        if g.account_id:
            annotation.account_id = g.account_id
        annotation.save()
        return jsonify(annotation.to_dict())
    else:
        return jsonify('No parameters given. Annotation not created.', status=400)

# READ
@store.route('/annotations/<id>')
def read_annotation(id):
    annotation = Annotation.get(id)

    if not annotation:
        return jsonify('Annotation not found.', status=404)
    elif authorize(annotation, 'read', get_current_userid()):
        return jsonify(annotation.to_dict())
    else:
        return jsonify('Could not authorise request. Read not allowed', status=401)

# UPDATE
@store.route('/annotations/<id>', methods=['PUT'])
def update_annotation(id):
    annotation = Annotation.get(id)

    if not annotation:
        return jsonify('Annotation not found. No update performed.', status=404)

    elif request.json and authorize(annotation, 'update', get_current_userid()):
        updated = Annotation.from_dict(request.json)
        if updated.permissions != annotation.permissions:
            if not authorize(annotation, ACTION.ADMIN, get_current_userid()):
                return jsonify('Could not authorise request (permissions change). No update performed', status=401)
        updated.save()
        return jsonify(updated.to_dict())
    else:
        return jsonify('Could not authorise request. No update performed', status=401)

# DELETE
@store.route('/annotations/<id>', methods=['DELETE'])
def delete_annotation(id):
    annotation = Annotation.get(id)

    if not annotation:
        return jsonify('Annotation not found. No delete performed.', status=404)

    elif authorize(annotation, 'delete', get_current_userid()):
        annotation.delete()
        return None, 204

    else:
        return jsonify('Could not authorise request. No update performed', status=401)

# Search
@store.route('/search')
def search_annotations():
    kwargs = dict(request.args.items())
    # TODO: limit results returned to max 200
    results = [ x.to_dict() for x in Annotation.search(**kwargs) ]
    # TODO: a proper count(*) for this query
    total = len(results)
    qrows = {
        'total': total,
        'rows': results,
    }
    return jsonify(qrows)

<|endoftext|>


<commit_before>#!/usr/bin/python
#coding:utf8

__author__ = ['markshao']

from multiprocessing import Process

def process_map(iter, target, **kwargs):
    # the different map implementation base on thread
    # def target(item,name):
    #     pass
    # iter = [item1,item2]
    # process_map(iter,target,**(dict(name="a"))

    process_list = []

    for item in iter:
        process = Process(target=target, args=[item], kwargs=kwargs)
        process.daemon = True
        process_list.append(process)
        process.start()

    for process in process_list:
        process.join()

if __name__ == "__main__":
    def test(item, bench):
        print "%s%s" % (item, bench)

    process_map(["mark", "archer"], test, **dict(bench="bbb"))<commit_msg>enhance the multiprocess handling<commit_after>#!/usr/bin/python
#coding:utf8

__author__ = ['markshao']

from multiprocessing import Process
from pagrant.exceptions import PagrantError

def process_map(iter, target, **kwargs):
    # the different map implementation base on thread
    # def target(item,name):
    #     pass
    # iter = [item1,item2]
    # process_map(iter,target,**(dict(name="a"))

    process_list = []

    for item in iter:
        process = Process(target=target, args=[item], kwargs=kwargs)
        process.daemon = True
        process_list.append(process)

    for process in process_list:
        process.start()

    for process in process_list:
        process.join()

    for process in process_list:
        if process.exitcode != 0:
            raise PagrantError("provision sub process get the exception")

if __name__ == "__main__":
    def test(item, bench):
        print "%s%s" % (item, bench)

    process_map(["mark", "archer"], test, **dict(bench="bbb"))<|endoftext|>