<commit_before>"""
Cross Compilation and RPC
=========================
**Author**: `Ziheng Jiang <https://github.com/ZihengJiang/>`_

This tutorial introduces cross compilation and remote device
execution with RPC in TVM.

With cross compilation and RPC, you can **compile program on your
local machine then run it on remote device**. It is useful when the
resource of remote device is limited, like Raspberry Pi and mobile
platforms, so you do not wish to put the compilation procedure on
the device in order to save time and space.
In this tutorial, I will take Raspberry Pi as our target platform
for example.
"""

######################################################################
# Build TVM Runtime on Device
# ---------------------------
#
# There're some prerequisites: similar as compiling TVM on your
# local machine, we need build runtime on remote device.
#
# .. note::
#
#   All instructions in both this section and next section should be
#   executed on the target device, e.g. Raspberry Pi. And we assume it
#   has Linux running.
#
# To get started, clone tvm repo from github. It is important to clone
# the submodules along, with --recursive option (Assuming you are in
# your home directory):
#
#   .. code-block:: bash
#
#     git clone --recursive https://github.com/dmlc/tvm
#
# .. note::
#
#   Usually device has limited resources and we only need to build
#   runtime. The idea is we will use TVM compiler on the local server
#   to compile and upload the compiled program to the device and run
#   the device function remotely.
#
#   .. code-block:: bash
#
#     cd tvm
#     cp make/config.mk .
#     echo USE_RPC=1>> config.mk
#
#   Also make sure that you have set :code:`USE_RPC=1` in your
#   :code:`config.mk`. We don't need LLVM when building runtime, so
#   :code:`LLVM_CONFIG = llvm-config` in :code:`config.mk` is commented
#   out by default. After that, build runtime!
#
#   .. code-block:: bash
#
#     make runtime
#
# After success of buildind runtime, we need set environment varibles
# in :code:`~/.bashrc` file of yourself account or :code:`/etc/profile`
# of system enviroment variables. Assuming your TVM directory is in
# :code:`~/tvm` and set environment variables below your account.
#
#   .. code-block:: bash
#
#    vi ~/.bashrc
#
# We need edit :code:`~/.bashrc` using :code:`vi ~/.bashrc` and add
# lines below (Assuming your TVM directory is in :code:`~/tvm`):
#
#   .. code-block:: bash
#
#    export TVM_HOME=~/tvm
#    export PATH=$PATH:$TVM_HOME/lib
#    export PYTHONPATH=$PYTHONPATH:$TVM_HOME/python
#
# To enable updated :code:`~/.bashrc`, execute :code:`source ~/.bashrc`.

######################################################################
# Set Up RPC Server on Device
# ---------------------------
# To set up a TVM RPC server on the Raspberry Pi (our remote device),
# we have prepared a one-line script so you only need to run this
# command after following the installation guide to install TVM on
# your device:
#
#   .. code-block:: bash
#
#     python -m tvm.exec.rpc_server --host 0.0.0.0 --port=9090
#
# After executing command above, if you see these lines below, it's
# successful to start RPC server on your device.
#
#    .. code-block:: bash
#
#      Loading runtime library /home/YOURNAME/code/tvm/lib/libtvm_runtime.so... exec only
#      INFO:root:RPCServer: bind to 0.0.0.0:9090
#
# In the following code block, we simply start an RPC server on the
# same machine, for demonstration. This line can be omitted if we
# started an remote server.
#
server = rpc.Server(host='0.0.0.0', port=9090, use_popen=True)

######################################################################
# Declare and Cross Compile Kernel on Local Machine
# -------------------------------------------------
#
# .. note::
#
#   Now we back to the local machine, which has a full TVM installed.
#
# Here we will declare a simple kernel with TVM on the local machine:

from __future__ import absolute_import, print_function

import tvm
import numpy as np
from tvm.contrib import rpc, util

n = tvm.convert(1024)
A = tvm.placeholder((n,), name='A')
B = tvm.compute(A.shape, lambda *i: A(*i) + 1.0, name='B')
s = tvm.create_schedule(B.op)

######################################################################
# Then we cross compile the kernel:
#

# the target here should be 'llvm -target=armv7l-none-linux-gnueabihf',
# and we use 'llvm' here to make example run locally, see the detailed
# note in the following block
f = tvm.build(s, [A, B], target='llvm', name='myadd')
# save the lib at local temp folder
temp = util.tempdir()
path = temp.relpath('mylib.o')
f.save(path)

######################################################################
# .. note::
#
#   the argument :code:`target` in :code:`build` should be replaced
#   :code:`'llvm'` with the target triple of your device, which might be
#   different for different device. For example, it is
#   :code:`'llvm -target=armv7l-none-linux-gnueabihf'` for my Raspberry
#   Pi. Here we use :code:`'llvm'` directly to make the tutorial runable.
#
#   Usually, you can query the target by execute :code:`gcc -v` on your
#   device, and look for the line starting with :code:`Target:`
#   (Though it may be still a loose configuration.)
#
#   Besides :code:`-target`, you can also set other compilation options
#   like:
#
#   * -mtriple=<target triple>
#       Specify the target triple, same as '-target'.
#   * -mcpu=<cpuname>
#       Specify a specific chip in the current architecture to generate code for. By default this is inferred from the target triple and autodetected to the current architecture.
#   * -mattr=a1,+a2,-a3,...
#       Override or control specific attributes of the target, such as whether SIMD operations are enabled or not. The default set of attributes is set by the current CPU.
#       To get the list of available attributes, you can do:
#
#       .. code-block:: bash
#
#         llc -mtriple=<your device target triple> -mattr=help
#
#   These options are consistent with `llc <http://llvm.org/docs/CommandGuide/llc.html>`_.
#   So for my board, to get the best performance, the complete compilation
#   option would be:
#
#   .. code-block:: bash
#
#     llvm -mtriple=armv7l-none-linux-gnueabihf -mcpu=cortex-a53 -mattr=+neon
#
#   It is recommended to set target triple and feature set to contain specific
#   feature available, so we can take full advantage of the features of the
#   board.
#   You can find more details about cross compilation attributes from
#   `LLVM guide of cross compilation <https://clang.llvm.org/docs/CrossCompilation.html>`_.

######################################################################
# Run CPU Kernel Remotely by RPC
# ------------------------------
# Here we will show you how to run the kernel on the remote device:
#
# .. note::
#    In order to have this tutorial runs locally to build the nice HTML, we
#    start a RPC server on the local machine. You can ignore it if you already
#    started the server on the target device. And then change host IP properly.

# Can be ignored if you already started the RPC server
server = rpc.Server(host='0.0.0.0', port=9090, use_popen=True)
host = '0.0.0.0'  # Change to your target device IP
port = 9090
# connect the remote device
remote = rpc.connect(host, port)

######################################################################
# Here we upload the lib to the remote device, then invoke a device local
# compiler for shared lib and load it into device memory. now `f` is a
# remote module object.
remote.upload(path)
f = remote.load_module('mylib.o')

# create array on the remote device
ctx = remote.cpu(0)
a = tvm.nd.array(np.random.uniform(size=1024).astype(A.dtype), ctx)
b = tvm.nd.array(np.zeros(1024, dtype=A.dtype), ctx)
# the function will run on the remote device
f(a, b)
np.testing.assert_equal(b.asnumpy(), a.asnumpy() + 1)

######################################################################
# When you want to evaluate the performance of the kernel on the remote
# device, it is important to avoid overhead of remote function call.
# :code:`time_evaluator` will returns a remote function that runs the
# function over number times, measures the cost per run on the remote
# device and returns the measured cost.
#
time_f = f.time_evaluator(f.entry_name, ctx, number=10)
cost = time_f(a, b).mean
print('%g secs/op' % cost)

#########################################################################
# Run OpenCL Kernel Remotely by RPC
# ---------------------------------
# As for remote OpenCL devices, the workflow is almost the same as above.
# You can define the kernel, upload files, and run by RPC. The files
# include host object, kernel source code and module meta file. We rely
# on remote compiler to re-link them.
#
# .. note::
#
#    Raspberry Pi does not support OpenCL, the following code is tested on
#    Firefly-RK3399. The target_host should be 'llvm -target=aarch64-linux-gnu'.
#    But here we set 'llvm' to enable this tutorial to run locally.
#
#    Also we need to build the runtime with the flag `USE_OPENCL=1`.

# build kernel (different from cpu, we need bind axis for OpenCL)
s = tvm.create_schedule(B.op)
xo, xi = s[B].split(B.op.axis[0], factor=32)
s[B].bind(xo, tvm.thread_axis("blockIdx.x"))
s[B].bind(xi, tvm.thread_axis("threadIdx.x"))
f = tvm.build(s, [A, B], "opencl", target_host="llvm", name="myadd")

# save files
path_o = temp.relpath("myadd.o")
path_cl = temp.relpath("myadd.cl")
path_json = temp.relpath("myadd.tvm_meta.json")
f.save(path_o)
f.imported_modules[0].save(path_cl)

# upload files
remote.upload(path_o)
remote.upload(path_cl)
remote.upload(path_json)

# load files on remote device
fhost = remote.load_module("myadd.o")
fdev = remote.load_module("myadd.cl")
fhost.import_module(fdev)

# run
ctx = remote.cl(0)
a = tvm.nd.array(np.random.uniform(size=1024).astype(A.dtype), ctx)
b = tvm.nd.array(np.zeros(1024, dtype=A.dtype), ctx)
fhost(a, b)
np.testing.assert_equal(b.asnumpy(), a.asnumpy() + 1)

#####################################################################
# Instead of uploading files separately, there is a more convinient way.
# You can export libraray as a tar ball.
path_tar = temp.relpath("myadd.tar")
f.export_library(path_tar)
remote.upload(path_tar)
fhost = remote.load_module("myadd.tar")
fhost(a, b)
np.testing.assert_equal(b.asnumpy(), a.asnumpy() + 1)

# terminate the server after experiment
server.terminate()

######################################################################
# Summary
# -------
# This tutorial provides a walk through of cross compilation and RPC
# features in TVM.
#
# - Set up RPC server on the remote device.
# - Set up target device configuration to cross compile kernel on the
#   local machine.
# - Upload and run the kernel remotely by RPC API.
<commit_msg>fix rpc tutorial (#818)<commit_after>"""
Cross Compilation and RPC
=========================
**Author**: `Ziheng Jiang <https://github.com/ZihengJiang/>`_

This tutorial introduces cross compilation and remote device
execution with RPC in TVM.

With cross compilation and RPC, you can **compile program on your
local machine then run it on remote device**. It is useful when the
resource of remote device is limited, like Raspberry Pi and mobile
platforms, so you do not wish to put the compilation procedure on
the device in order to save time and space.
In this tutorial, I will take Raspberry Pi as our target platform
for example.
"""

######################################################################
# Build TVM Runtime on Device
# ---------------------------
#
# There're some prerequisites: similar as compiling TVM on your
# local machine, we need build runtime on remote device.
#
# .. note::
#
#   All instructions in both this section and next section should be
#   executed on the target device, e.g. Raspberry Pi. And we assume it
#   has Linux running.
#
# To get started, clone tvm repo from github. It is important to clone
# the submodules along, with --recursive option (Assuming you are in
# your home directory):
#
#   .. code-block:: bash
#
#     git clone --recursive https://github.com/dmlc/tvm
#
# .. note::
#
#   Usually device has limited resources and we only need to build
#   runtime. The idea is we will use TVM compiler on the local server
#   to compile and upload the compiled program to the device and run
#   the device function remotely.
#
#   .. code-block:: bash
#
#     cd tvm
#     cp make/config.mk .
#     echo USE_RPC=1>> config.mk
#
#   Also make sure that you have set :code:`USE_RPC=1` in your
#   :code:`config.mk`. We don't need LLVM when building runtime, so
#   :code:`LLVM_CONFIG = llvm-config` in :code:`config.mk` is commented
#   out by default. After that, build runtime!
#
#   .. code-block:: bash
#
#     make runtime
#
# After success of buildind runtime, we need set environment varibles
# in :code:`~/.bashrc` file of yourself account or :code:`/etc/profile`
# of system enviroment variables. Assuming your TVM directory is in
# :code:`~/tvm` and set environment variables below your account.
#
#   .. code-block:: bash
#
#    vi ~/.bashrc
#
# We need edit :code:`~/.bashrc` using :code:`vi ~/.bashrc` and add
# lines below (Assuming your TVM directory is in :code:`~/tvm`):
#
#   .. code-block:: bash
#
#    export TVM_HOME=~/tvm
#    export PATH=$PATH:$TVM_HOME/lib
#    export PYTHONPATH=$PYTHONPATH:$TVM_HOME/python
#
# To enable updated :code:`~/.bashrc`, execute :code:`source ~/.bashrc`.

######################################################################
# Set Up RPC Server on Device
# ---------------------------
# To set up a TVM RPC server on the Raspberry Pi (our remote device),
# we have prepared a one-line script so you only need to run this
# command after following the installation guide to install TVM on
# your device:
#
#   .. code-block:: bash
#
#     python -m tvm.exec.rpc_server --host 0.0.0.0 --port=9090
#
# After executing command above, if you see these lines below, it's
# successful to start RPC server on your device.
#
#    .. code-block:: bash
#
#      Loading runtime library /home/YOURNAME/code/tvm/lib/libtvm_runtime.so... exec only
#      INFO:root:RPCServer: bind to 0.0.0.0:9090
#
# In the following code block, we simply start an RPC server on the
# same machine, for demonstration. This line can be omitted if we
# started an remote server.
#
from __future__ import absolute_import, print_function

import tvm
import numpy as np
from tvm.contrib import rpc, util

server = rpc.Server(host='0.0.0.0', port=9090, use_popen=True)

######################################################################
# Declare and Cross Compile Kernel on Local Machine
# -------------------------------------------------
#
# .. note::
#
#   Now we back to the local machine, which has a full TVM installed.
#
# Here we will declare a simple kernel with TVM on the local machine:

n = tvm.convert(1024)
A = tvm.placeholder((n,), name='A')
B = tvm.compute(A.shape, lambda *i: A(*i) + 1.0, name='B')
s = tvm.create_schedule(B.op)

######################################################################
# Then we cross compile the kernel:
#

# the target here should be 'llvm -target=armv7l-none-linux-gnueabihf',
# and we use 'llvm' here to make example run locally, see the detailed
# note in the following block
f = tvm.build(s, [A, B], target='llvm', name='myadd')
# save the lib at local temp folder
temp = util.tempdir()
path = temp.relpath('mylib.o')
f.save(path)

######################################################################
# .. note::
#
#   the argument :code:`target` in :code:`build` should be replaced
#   :code:`'llvm'` with the target triple of your device, which might be
#   different for different device. For example, it is
#   :code:`'llvm -target=armv7l-none-linux-gnueabihf'` for my Raspberry
#   Pi. Here we use :code:`'llvm'` directly to make the tutorial runable.
#
#   Usually, you can query the target by execute :code:`gcc -v` on your
#   device, and look for the line starting with :code:`Target:`
#   (Though it may be still a loose configuration.)
#
#   Besides :code:`-target`, you can also set other compilation options
#   like:
#
#   * -mtriple=<target triple>
#       Specify the target triple, same as '-target'.
#   * -mcpu=<cpuname>
#       Specify a specific chip in the current architecture to generate code for. By default this is inferred from the target triple and autodetected to the current architecture.
#   * -mattr=a1,+a2,-a3,...
#       Override or control specific attributes of the target, such as whether SIMD operations are enabled or not. The default set of attributes is set by the current CPU.
#       To get the list of available attributes, you can do:
#
#       .. code-block:: bash
#
#         llc -mtriple=<your device target triple> -mattr=help
#
#   These options are consistent with `llc <http://llvm.org/docs/CommandGuide/llc.html>`_.
#   So for my board, to get the best performance, the complete compilation
#   option would be:
#
#   .. code-block:: bash
#
#     llvm -mtriple=armv7l-none-linux-gnueabihf -mcpu=cortex-a53 -mattr=+neon
#
#   It is recommended to set target triple and feature set to contain specific
#   feature available, so we can take full advantage of the features of the
#   board.
#   You can find more details about cross compilation attributes from
#   `LLVM guide of cross compilation <https://clang.llvm.org/docs/CrossCompilation.html>`_.

######################################################################
# Run CPU Kernel Remotely by RPC
# ------------------------------
# Here we will show you how to run the kernel on the remote device:
#
# .. note::
#    In order to have this tutorial runs locally to build the nice HTML, we
#    start a RPC server on the local machine. You can ignore it if you already
#    started the server on the target device. And then change host IP properly.

# Can be ignored if you already started the RPC server
server = rpc.Server(host='0.0.0.0', port=9090, use_popen=True)
host = '0.0.0.0'  # Change to your target device IP
port = 9090
# connect the remote device
remote = rpc.connect(host, port)

######################################################################
# Here we upload the lib to the remote device, then invoke a device local
# compiler for shared lib and load it into device memory. now `f` is a
# remote module object.
remote.upload(path)
f = remote.load_module('mylib.o')

# create array on the remote device
ctx = remote.cpu(0)
a = tvm.nd.array(np.random.uniform(size=1024).astype(A.dtype), ctx)
b = tvm.nd.array(np.zeros(1024, dtype=A.dtype), ctx)
# the function will run on the remote device
f(a, b)
np.testing.assert_equal(b.asnumpy(), a.asnumpy() + 1)

######################################################################
# When you want to evaluate the performance of the kernel on the remote
# device, it is important to avoid overhead of remote function call.
# :code:`time_evaluator` will returns a remote function that runs the
# function over number times, measures the cost per run on the remote
# device and returns the measured cost.
#
time_f = f.time_evaluator(f.entry_name, ctx, number=10)
cost = time_f(a, b).mean
print('%g secs/op' % cost)

#########################################################################
# Run OpenCL Kernel Remotely by RPC
# ---------------------------------
# As for remote OpenCL devices, the workflow is almost the same as above.
# You can define the kernel, upload files, and run by RPC. The files
# include host object, kernel source code and module meta file. We rely
# on remote compiler to re-link them.
#
# .. note::
#
#    Raspberry Pi does not support OpenCL, the following code is tested on
#    Firefly-RK3399. The target_host should be 'llvm -target=aarch64-linux-gnu'.
#    But here we set 'llvm' to enable this tutorial to run locally.
#
#    Also we need to build the runtime with the flag `USE_OPENCL=1`.

# build kernel (different from cpu, we need bind axis for OpenCL)
s = tvm.create_schedule(B.op)
xo, xi = s[B].split(B.op.axis[0], factor=32)
s[B].bind(xo, tvm.thread_axis("blockIdx.x"))
s[B].bind(xi, tvm.thread_axis("threadIdx.x"))
f = tvm.build(s, [A, B], "opencl", target_host="llvm", name="myadd")

# save files
path_o = temp.relpath("myadd.o")
path_cl = temp.relpath("myadd.cl")
path_json = temp.relpath("myadd.tvm_meta.json")
f.save(path_o)
f.imported_modules[0].save(path_cl)

# upload files
remote.upload(path_o)
remote.upload(path_cl)
remote.upload(path_json)

# load files on remote device
fhost = remote.load_module("myadd.o")
fdev = remote.load_module("myadd.cl")
fhost.import_module(fdev)

# run
ctx = remote.cl(0)
a = tvm.nd.array(np.random.uniform(size=1024).astype(A.dtype), ctx)
b = tvm.nd.array(np.zeros(1024, dtype=A.dtype), ctx)
fhost(a, b)
np.testing.assert_equal(b.asnumpy(), a.asnumpy() + 1)

#####################################################################
# Instead of uploading files separately, there is a more convinient way.
# You can export libraray as a tar ball.
path_tar = temp.relpath("myadd.tar")
f.export_library(path_tar)
remote.upload(path_tar)
fhost = remote.load_module("myadd.tar")
fhost(a, b)
np.testing.assert_equal(b.asnumpy(), a.asnumpy() + 1)

# terminate the server after experiment
server.terminate()

######################################################################
# Summary
# -------
# This tutorial provides a walk through of cross compilation and RPC
# features in TVM.
#
# - Set up RPC server on the remote device.
# - Set up target device configuration to cross compile kernel on the
#   local machine.
# - Upload and run the kernel remotely by RPC API.
<|endoftext|>


<commit_before>#! /usr/bin/env python
# -*- coding: utf-8  -*-
#
# Copyright (C) 2009-2012 Ben Kurtovic <ben.kurtovic@verizon.net>
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

import imp
from os import listdir, path
from re import sub
from threading import RLock, Thread
from time import gmtime, strftime

from earwigbot.commands import Command
from earwigbot.tasks import Task

__all__ = ["CommandManager", "TaskManager"]

class _ResourceManager(object):
    """
    **EarwigBot: Resource Manager**

    Resources are essentially objects dynamically loaded by the bot, both
    packaged with it (built-in resources) and created by users (plugins, aka
    custom resources). Currently, the only two types of resources are IRC
    commands and bot tasks. These are both loaded from two locations: the
    :py:mod:`earwigbot.commands` and :py:mod:`earwigbot.tasks packages`, and
    the :file:`commands/` and :file:`tasks/` directories within the bot's
    working directory.

    This class handles the low-level tasks of (re)loading resources via
    :py:meth:`load`, retrieving specific resources via :py:meth:`get`, and
    iterating over all resources via :py:meth:`__iter__`.
    """
    def __init__(self, bot, name, base):
        self.bot = bot
        self.logger = bot.logger.getChild(name)

        self._resources = {}
        self._resource_name = name  # e.g. "commands" or "tasks"
        self._resource_base = base  # e.g. Command or Task
        self._resource_access_lock = RLock()

    def __repr__(self):
        """Return the canonical string representation of the manager."""
        res = "{0}(bot={1!r}, name={2!r}, base={3!r})"
        return res.format(self.__class__.__name__, self.bot,
                          self._resource_name, self._resource_base)

    def __str__(self):
        """Return a nice string representation of the manager."""
        return "<{0} of {1}>".format(self.__class__.__name__, self.bot)

    def __iter__(self):
        with self.lock:
            for resource in self._resources.itervalues():
                yield resource

    def _load_resource(self, name, path, klass):
        """Instantiate a resource class and add it to the dictionary."""
        res_type = self._resource_name[:-1]  # e.g. "command" or "task"
        if hasattr(klass, "name"):
            res_config = getattr(self.bot.config, self._resource_name)
            if getattr(klass, "name") in res_config.get("disable", []):
                log = "Skipping disabled {0} {1}"
                self.logger.debug(log.format(res_type, getattr(klass, "name")))
                return
        try:
            resource = klass(self.bot)  # Create instance of resource
        except Exception:
            e = "Error instantiating {0} class in '{1}' (from {2})"
            self.logger.exception(e.format(res_type, name, path))
        else:
            self._resources[resource.name] = resource
            self.logger.debug("Loaded {0} {1}".format(res_type, resource.name))

    def _load_module(self, name, path):
        """Load a specific resource from a module, identified by name and path.

        We'll first try to import it using imp magic, and if that works, make
        instances of any classes inside that are subclasses of the base
        (:py:attr:`self._resource_base <_resource_base>`), add them to the
        resources dictionary with :py:meth:`self._load_resource()
        <_load_resource>`, and finally log the addition. Any problems along
        the way will either be ignored or logged.
        """
        f, path, desc = imp.find_module(name, [path])
        try:
            module = imp.load_module(name, f, path, desc)
        except Exception:
            e = "Couldn't load module '{0}' (from {1})"
            self.logger.exception(e.format(name, path))
            return
        finally:
            f.close()

        for obj in vars(module).values():
            if type(obj) is type:
                isresource = issubclass(obj, self._resource_base)
                if isresource and not obj is self._resource_base:
                    self._load_resource(name, path, obj)

    def _load_directory(self, dir):
        """Load all valid resources in a given directory."""
        self.logger.debug("Loading directory {0}".format(dir))
        res_config = getattr(self.bot.config, self._resource_name)
        disabled = res_config.get("disable", [])
        processed = []
        for name in listdir(dir):
            if not name.endswith(".py") and not name.endswith(".pyc"):
                continue
            if name.startswith("_") or name.startswith("."):
                continue
            modname = sub("\.pyc?$", "", name)  # Remove extension
            if modname in disabled:
                log = "Skipping disabled module {0}".format(modname)
                self.logger.debug(log)
                continue
            if modname not in processed:
                self._load_module(modname, dir)
                processed.append(modname)

    @property
    def lock(self):
        """The resource access/modify lock."""
        return self._resource_access_lock

    def load(self):
        """Load (or reload) all valid resources into :py:attr:`_resources`."""
        name = self._resource_name  # e.g. "commands" or "tasks"
        with self.lock:
            self._resources.clear()
            builtin_dir = path.join(path.dirname(__file__), name)
            plugins_dir = path.join(self.bot.config.root_dir, name)
            if getattr(self.bot.config, name).get("disable") is True:
                log = "Skipping disabled builtins directory {0}"
                self.logger.debug(log.format(builtin_dir))
            else:
                self._load_directory(builtin_dir)  # Built-in resources
            self._load_directory(plugins_dir)  # Custom resources, aka plugins

        if self._resources:
            msg = "Loaded {0} {1}: {2}"
            resources = ", ".join(self._resources.keys())
            self.logger.info(msg.format(len(self._resources), name, resources))
        else:
            self.logger.info("Loaded 0 {0}".format(name))

    def get(self, key):
        """Return the class instance associated with a certain resource.

        Will raise :py:exc:`KeyError` if the resource (a command or task) is
        not found.
        """
        with self.lock:
            return self._resources[key]

class CommandManager(_ResourceManager):
    """
    Manages (i.e., loads, reloads, and calls) IRC commands.
    """
    def __init__(self, bot):
        super(CommandManager, self).__init__(bot, "commands", Command)

    def _wrap_check(self, command, data):
        """Check whether a command should be called, catching errors."""
        try:
            return command.check(data)
        except Exception:
            e = "Error checking command '{0}' with data: {1}:"
            self.logger.exception(e.format(command.name, data))

    def _wrap_process(self, command, data):
        """process() the message, catching and reporting any errors."""
        try:
            command.process(data)
        except Exception:
            e = "Error executing command '{0}':"
            self.logger.exception(e.format(command.name))

    def call(self, hook, data):
        """Respond to a hook type and a :py:class:`Data` object."""
        for command in self:
            if hook in command.hooks and self._wrap_check(command, data):
                thread = Thread(target=self._wrap_process,
                                args=(command, data))
                start_time = strftime("%b %d %H:%M:%S")
                thread.name = "irc:{0} ({1})".format(command.name, start_time)
                thread.daemon = True
                thread.start()
                return

class TaskManager(_ResourceManager):
    """
    Manages (i.e., loads, reloads, schedules, and runs) wiki bot tasks.
    """
    def __init__(self, bot):
        super(TaskManager, self).__init__(bot, "tasks", Task)

    def _wrapper(self, task, **kwargs):
        """Wrapper for task classes: run the task and catch any errors."""
        try:
            task.run(**kwargs)
        except Exception:
            msg = "Task '{0}' raised an exception and had to stop:"
            self.logger.exception(msg.format(task.name))
        else:
            msg = "Task '{0}' finished successfully"
            self.logger.info(msg.format(task.name))

    def start(self, task_name, **kwargs):
        """Start a given task in a new daemon thread, and return the thread.

        kwargs are passed to :py:meth:`task.run() <earwigbot.tasks.Task.run>`.
        If the task is not found, ``None`` will be returned an an error is
        logged.
        """
        msg = "Starting task '{0}' in a new thread"
        self.logger.info(msg.format(task_name))

        try:
            task = self.get(task_name)
        except KeyError:
            e = "Couldn't find task '{0}'"
            self.logger.error(e.format(task_name))
            return

        task_thread = Thread(target=self._wrapper, args=(task,), kwargs=kwargs)
        start_time = strftime("%b %d %H:%M:%S")
        task_thread.name = "{0} ({1})".format(task_name, start_time)
        task_thread.daemon = True
        task_thread.start()
        return task_thread

    def schedule(self, now=None):
        """Start all tasks that are supposed to be run at a given time."""
        if not now:
            now = gmtime()
        # Get list of tasks to run this turn:
        tasks = self.bot.config.schedule(now.tm_min, now.tm_hour, now.tm_mday,
                                         now.tm_mon, now.tm_wday)

        for task in tasks:
            if isinstance(task, list):          # They've specified kwargs,
                self.start(task[0], **task[1])  # so pass those to start
            else:  # Otherwise, just pass task_name
                self.start(task)
<commit_msg>Fix typo in TaskManager.start.__doc__<commit_after>#! /usr/bin/env python
# -*- coding: utf-8  -*-
#
# Copyright (C) 2009-2012 Ben Kurtovic <ben.kurtovic@verizon.net>
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

import imp
from os import listdir, path
from re import sub
from threading import RLock, Thread
from time import gmtime, strftime

from earwigbot.commands import Command
from earwigbot.tasks import Task

__all__ = ["CommandManager", "TaskManager"]

class _ResourceManager(object):
    """
    **EarwigBot: Resource Manager**

    Resources are essentially objects dynamically loaded by the bot, both
    packaged with it (built-in resources) and created by users (plugins, aka
    custom resources). Currently, the only two types of resources are IRC
    commands and bot tasks. These are both loaded from two locations: the
    :py:mod:`earwigbot.commands` and :py:mod:`earwigbot.tasks packages`, and
    the :file:`commands/` and :file:`tasks/` directories within the bot's
    working directory.

    This class handles the low-level tasks of (re)loading resources via
    :py:meth:`load`, retrieving specific resources via :py:meth:`get`, and
    iterating over all resources via :py:meth:`__iter__`.
    """
    def __init__(self, bot, name, base):
        self.bot = bot
        self.logger = bot.logger.getChild(name)

        self._resources = {}
        self._resource_name = name  # e.g. "commands" or "tasks"
        self._resource_base = base  # e.g. Command or Task
        self._resource_access_lock = RLock()

    def __repr__(self):
        """Return the canonical string representation of the manager."""
        res = "{0}(bot={1!r}, name={2!r}, base={3!r})"
        return res.format(self.__class__.__name__, self.bot,
                          self._resource_name, self._resource_base)

    def __str__(self):
        """Return a nice string representation of the manager."""
        return "<{0} of {1}>".format(self.__class__.__name__, self.bot)

    def __iter__(self):
        with self.lock:
            for resource in self._resources.itervalues():
                yield resource

    def _load_resource(self, name, path, klass):
        """Instantiate a resource class and add it to the dictionary."""
        res_type = self._resource_name[:-1]  # e.g. "command" or "task"
        if hasattr(klass, "name"):
            res_config = getattr(self.bot.config, self._resource_name)
            if getattr(klass, "name") in res_config.get("disable", []):
                log = "Skipping disabled {0} {1}"
                self.logger.debug(log.format(res_type, getattr(klass, "name")))
                return
        try:
            resource = klass(self.bot)  # Create instance of resource
        except Exception:
            e = "Error instantiating {0} class in '{1}' (from {2})"
            self.logger.exception(e.format(res_type, name, path))
        else:
            self._resources[resource.name] = resource
            self.logger.debug("Loaded {0} {1}".format(res_type, resource.name))

    def _load_module(self, name, path):
        """Load a specific resource from a module, identified by name and path.

        We'll first try to import it using imp magic, and if that works, make
        instances of any classes inside that are subclasses of the base
        (:py:attr:`self._resource_base <_resource_base>`), add them to the
        resources dictionary with :py:meth:`self._load_resource()
        <_load_resource>`, and finally log the addition. Any problems along
        the way will either be ignored or logged.
        """
        f, path, desc = imp.find_module(name, [path])
        try:
            module = imp.load_module(name, f, path, desc)
        except Exception:
            e = "Couldn't load module '{0}' (from {1})"
            self.logger.exception(e.format(name, path))
            return
        finally:
            f.close()

        for obj in vars(module).values():
            if type(obj) is type:
                isresource = issubclass(obj, self._resource_base)
                if isresource and not obj is self._resource_base:
                    self._load_resource(name, path, obj)

    def _load_directory(self, dir):
        """Load all valid resources in a given directory."""
        self.logger.debug("Loading directory {0}".format(dir))
        res_config = getattr(self.bot.config, self._resource_name)
        disabled = res_config.get("disable", [])
        processed = []
        for name in listdir(dir):
            if not name.endswith(".py") and not name.endswith(".pyc"):
                continue
            if name.startswith("_") or name.startswith("."):
                continue
            modname = sub("\.pyc?$", "", name)  # Remove extension
            if modname in disabled:
                log = "Skipping disabled module {0}".format(modname)
                self.logger.debug(log)
                continue
            if modname not in processed:
                self._load_module(modname, dir)
                processed.append(modname)

    @property
    def lock(self):
        """The resource access/modify lock."""
        return self._resource_access_lock

    def load(self):
        """Load (or reload) all valid resources into :py:attr:`_resources`."""
        name = self._resource_name  # e.g. "commands" or "tasks"
        with self.lock:
            self._resources.clear()
            builtin_dir = path.join(path.dirname(__file__), name)
            plugins_dir = path.join(self.bot.config.root_dir, name)
            if getattr(self.bot.config, name).get("disable") is True:
                log = "Skipping disabled builtins directory {0}"
                self.logger.debug(log.format(builtin_dir))
            else:
                self._load_directory(builtin_dir)  # Built-in resources
            self._load_directory(plugins_dir)  # Custom resources, aka plugins

        if self._resources:
            msg = "Loaded {0} {1}: {2}"
            resources = ", ".join(self._resources.keys())
            self.logger.info(msg.format(len(self._resources), name, resources))
        else:
            self.logger.info("Loaded 0 {0}".format(name))

    def get(self, key):
        """Return the class instance associated with a certain resource.

        Will raise :py:exc:`KeyError` if the resource (a command or task) is
        not found.
        """
        with self.lock:
            return self._resources[key]

class CommandManager(_ResourceManager):
    """
    Manages (i.e., loads, reloads, and calls) IRC commands.
    """
    def __init__(self, bot):
        super(CommandManager, self).__init__(bot, "commands", Command)

    def _wrap_check(self, command, data):
        """Check whether a command should be called, catching errors."""
        try:
            return command.check(data)
        except Exception:
            e = "Error checking command '{0}' with data: {1}:"
            self.logger.exception(e.format(command.name, data))

    def _wrap_process(self, command, data):
        """process() the message, catching and reporting any errors."""
        try:
            command.process(data)
        except Exception:
            e = "Error executing command '{0}':"
            self.logger.exception(e.format(command.name))

    def call(self, hook, data):
        """Respond to a hook type and a :py:class:`Data` object."""
        for command in self:
            if hook in command.hooks and self._wrap_check(command, data):
                thread = Thread(target=self._wrap_process,
                                args=(command, data))
                start_time = strftime("%b %d %H:%M:%S")
                thread.name = "irc:{0} ({1})".format(command.name, start_time)
                thread.daemon = True
                thread.start()
                return

class TaskManager(_ResourceManager):
    """
    Manages (i.e., loads, reloads, schedules, and runs) wiki bot tasks.
    """
    def __init__(self, bot):
        super(TaskManager, self).__init__(bot, "tasks", Task)

    def _wrapper(self, task, **kwargs):
        """Wrapper for task classes: run the task and catch any errors."""
        try:
            task.run(**kwargs)
        except Exception:
            msg = "Task '{0}' raised an exception and had to stop:"
            self.logger.exception(msg.format(task.name))
        else:
            msg = "Task '{0}' finished successfully"
            self.logger.info(msg.format(task.name))

    def start(self, task_name, **kwargs):
        """Start a given task in a new daemon thread, and return the thread.

        kwargs are passed to :py:meth:`task.run() <earwigbot.tasks.Task.run>`.
        If the task is not found, ``None`` will be returned and an error will
        be logged.
        """
        msg = "Starting task '{0}' in a new thread"
        self.logger.info(msg.format(task_name))

        try:
            task = self.get(task_name)
        except KeyError:
            e = "Couldn't find task '{0}'"
            self.logger.error(e.format(task_name))
            return

        task_thread = Thread(target=self._wrapper, args=(task,), kwargs=kwargs)
        start_time = strftime("%b %d %H:%M:%S")
        task_thread.name = "{0} ({1})".format(task_name, start_time)
        task_thread.daemon = True
        task_thread.start()
        return task_thread

    def schedule(self, now=None):
        """Start all tasks that are supposed to be run at a given time."""
        if not now:
            now = gmtime()
        # Get list of tasks to run this turn:
        tasks = self.bot.config.schedule(now.tm_min, now.tm_hour, now.tm_mday,
                                         now.tm_mon, now.tm_wday)

        for task in tasks:
            if isinstance(task, list):          # They've specified kwargs,
                self.start(task[0], **task[1])  # so pass those to start
            else:  # Otherwise, just pass task_name
                self.start(task)
<|endoftext|>


<commit_before>#!/usr/bin/env python
# encoding: utf-8

"""Pyvona : an IVONA python library
Author: Zachary Bears
Contact Email: bears.zachary@gmail.com
Note: Full operation of this library requires the requests and pygame libraries
"""

import datetime
import hashlib
import hmac
import json
import tempfile

class PyvonaException(Exception):
    pass

try:
    import pygame
except ImportError:
    pygame_available = False
else:
    pygame_available = True

try:
    import requests
except ImportError:
    msg = 'The requests library is essential for Pyvona operation. '
    msg += 'Without it, Pyvona will not function correctly.'
    raise PyvonaException(msg)

_amazon_date_format = '%Y%m%dT%H%M%SZ'
_date_format = '%Y%m%d'

def create_voice(access_key, secret_key):
    """Creates and returns a voice object to interact with
    """
    return Voice(access_key, secret_key)

class Voice(object):

    """An object that contains all the required methods for interacting
    with the IVONA text-to-speech system
    """
    voice_name = None
    speech_rate = None
    sentence_break = None
    paragraph_break = None
    _codec = "ogg"
    region_options = {
        'us-east': 'us-east-1',
        'us-west': 'us-west-2',
        'eu-west': 'eu-west-1',
    }
    access_key = None
    secret_key = None

    algorithm = 'AWS4-HMAC-SHA256'
    signed_headers = 'content-type;host;x-amz-content-sha256;x-amz-date'
    _region = None
    _host = None

    @property
    def region(self):
        return self._region

    @region.setter
    def region(self, region_name):
        self._region = self.region_options.get(region_name, 'us-east-1')
        self._host = 'tts.{}.ivonacloud.com'.format(self._region)

    @property
    def codec(self):
        return self._codec

    @codec.setter
    def codec(self, codec):
        if codec not in ["mp3", "ogg"]:
            raise PyvonaException(
                "Invalid codec specified. Please choose 'mp3' or 'ogg'")
        self._codec = codec

    def fetch_voice_ogg(self, text_to_speak, filename):
        """Fetch an ogg file for given text and save it to the given file name
        """
        current_codec = self.codec
        self.codec = "ogg"
        self.fetch_voice(text_to_speak, filename)
        self.codec = current_codec

    def fetch_voice(self, text_to_speak, filename):
        """Fetch a voice file for given text and save it to the given file name
        """
        file_extension = ".{codec}".format(codec=self.codec)
        filename += file_extension if not filename.endswith(
            file_extension) else ""
        with open(filename, 'wb') as f:
            self.fetch_voice_fp(text_to_speak, f)

    def fetch_voice_fp(self, text_to_speak, fp):
        """Fetch a voice file for given text and save it to the given file pointer
        """
        r = self._send_amazon_auth_packet_v4(
            'POST', 'tts', 'application/json', '/CreateSpeech', '',
            self._generate_payload(text_to_speak), self._region, self._host)
        if r.content.startswith(b'{'):
            raise PyvonaException('Error fetching voice: {}'.format(r.content))
        else:
            fp.write(r.content)

    def speak(self, text_to_speak):
        """Speak a given text
        """
        if not pygame_available:
            raise PyvonaException(
                "Pygame not installed. Please install to use speech.")

        with tempfile.SpooledTemporaryFile() as f:
            current_codec = self.codec
            self.codec = "ogg"
            self.fetch_voice_fp(text_to_speak, f)
            self.codec = current_codec
            f.seek(0)
            channel = pygame.mixer.Channel(5)
            sound = pygame.mixer.Sound(f)
            channel.play(sound)
            while channel.get_busy():
                pass

    def list_voices(self):
        """Returns all the possible voices
        """
        r = self._send_amazon_auth_packet_v4(
            'POST', 'tts', 'application/json', '/ListVoices', '', '',
            self._region, self._host)
        return r.content

    def _generate_payload(self, text_to_speak):
        return json.dumps({
            'Input': {
                'Data': text_to_speak
            },
            'OutputFormat': {
                'Codec': self.codec.upper()
            },
            'Parameters': {
                'Rate': self.speech_rate,
                'SentenceBreak': self.sentence_break,
                'ParagraphBreak': self.paragraph_break
            },
            'Voice': {
                'Name': self.voice_name
            }
        })

    def _send_amazon_auth_packet_v4(self, method, service, content_type,
                                    canonical_uri, canonical_querystring,
                                    request_parameters, region, host):
        """Send a packet to a given amazon server using Amazon's signature Version 4,
        Returns the resulting response object
        """
        # Create date for headers and the credential string
        t = datetime.datetime.utcnow()
        amazon_date = t.strftime(_amazon_date_format)
        date_stamp = t.strftime(_date_format)

        # Step 1: Create canonical request
        payload_hash = self._sha_hash(request_parameters)

        canonical_headers = 'content-type:{}\n'.format(content_type)
        canonical_headers += 'host:{}\n'.format(host)
        canonical_headers += 'x-amz-content-sha256:{}\n'.format(payload_hash)
        canonical_headers += 'x-amz-date:{}\n'.format(amazon_date)

        canonical_request = '\n'.join([
            method, canonical_uri, canonical_querystring, canonical_headers,
            self.signed_headers, payload_hash])

        # Step 2: Create the string to sign
        credential_scope = '{}/{}/{}/aws4_request'.format(
            date_stamp, region, service)
        string_to_sign = '\n'.join([
            self.algorithm, amazon_date, credential_scope,
            self._sha_hash(canonical_request)])

        # Step 3: Calculate the signature
        signing_key = self._get_signature_key(
            self.secret_key, date_stamp, region, service)
        signature = hmac.new(
            signing_key, string_to_sign.encode('utf-8'),
            hashlib.sha256).hexdigest()

        # Step 4: Create the signed packet
        endpoint = 'https://{}{}'.format(host, canonical_uri)
        authorization_header = '{} Credential={}/{}, ' +\
            'SignedHeaders={}, Signature={}'
        authorization_header = authorization_header.format(
            self.algorithm, self.access_key, credential_scope,
            self.signed_headers, signature)
        headers = {
            'Host': host,
            'Content-type': content_type,
            'X-Amz-Date': amazon_date,
            'Authorization': authorization_header,
            'x-amz-content-sha256': payload_hash,
            'Content-Length': len(request_parameters)
        }
        # Send the packet and return the response
        return requests.post(endpoint, data=request_parameters,
                             headers=headers)

    def _sha_hash(self, to_hash):
        return hashlib.sha256(to_hash.encode('utf-8')).hexdigest()

    def _sign(self, key, msg):
        return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()

    def _get_signature_key(self, key, date_stamp, region_name, service_name):
        k_date = self._sign(('AWS4{}'.format(key)).encode('utf-8'), date_stamp)
        k_region = self._sign(k_date, region_name)
        k_service = self._sign(k_region, service_name)
        k_signing = self._sign(k_service, 'aws4_request')
        return k_signing

    def __init__(self, access_key, secret_key):
        """Set initial voice object parameters
        """
        self.region = 'us-east'
        self.voice_name = 'Brian'
        self.access_key = access_key
        self.secret_key = secret_key
        self.speech_rate = 'medium'
        self.sentence_break = 400
        self.paragraph_break = 650
        if pygame_available:
            pygame.mixer.init()
<commit_msg>Implement use_ogg_codec contextmanager<commit_after>#!/usr/bin/env python
# encoding: utf-8

"""Pyvona : an IVONA python library
Author: Zachary Bears
Contact Email: bears.zachary@gmail.com
Note: Full operation of this library requires the requests and pygame libraries
"""

import datetime
import hashlib
import hmac
import json
import tempfile
import contextlib

class PyvonaException(Exception):
    pass

try:
    import pygame
except ImportError:
    pygame_available = False
else:
    pygame_available = True

try:
    import requests
except ImportError:
    msg = 'The requests library is essential for Pyvona operation. '
    msg += 'Without it, Pyvona will not function correctly.'
    raise PyvonaException(msg)

_amazon_date_format = '%Y%m%dT%H%M%SZ'
_date_format = '%Y%m%d'

def create_voice(access_key, secret_key):
    """Creates and returns a voice object to interact with
    """
    return Voice(access_key, secret_key)

class Voice(object):

    """An object that contains all the required methods for interacting
    with the IVONA text-to-speech system
    """
    voice_name = None
    speech_rate = None
    sentence_break = None
    paragraph_break = None
    _codec = "ogg"
    region_options = {
        'us-east': 'us-east-1',
        'us-west': 'us-west-2',
        'eu-west': 'eu-west-1',
    }
    access_key = None
    secret_key = None

    algorithm = 'AWS4-HMAC-SHA256'
    signed_headers = 'content-type;host;x-amz-content-sha256;x-amz-date'
    _region = None
    _host = None

    @property
    def region(self):
        return self._region

    @region.setter
    def region(self, region_name):
        self._region = self.region_options.get(region_name, 'us-east-1')
        self._host = 'tts.{}.ivonacloud.com'.format(self._region)

    @property
    def codec(self):
        return self._codec

    @codec.setter
    def codec(self, codec):
        if codec not in ["mp3", "ogg"]:
            raise PyvonaException(
                "Invalid codec specified. Please choose 'mp3' or 'ogg'")
        self._codec = codec

    @contextlib.contextmanager
    def use_ogg_codec(self):
        current_codec = self.codec
        self.codec = "ogg"
        try:
            yield
        finally:
            self.codec = current_codec

    def fetch_voice_ogg(self, text_to_speak, filename):
        """Fetch an ogg file for given text and save it to the given file name
        """
        with self.use_ogg_codec():
            self.fetch_voice(text_to_speak, filename)

    def fetch_voice(self, text_to_speak, filename):
        """Fetch a voice file for given text and save it to the given file name
        """
        file_extension = ".{codec}".format(codec=self.codec)
        filename += file_extension if not filename.endswith(
            file_extension) else ""
        with open(filename, 'wb') as f:
            self.fetch_voice_fp(text_to_speak, f)

    def fetch_voice_fp(self, text_to_speak, fp):
        """Fetch a voice file for given text and save it to the given file pointer
        """
        r = self._send_amazon_auth_packet_v4(
            'POST', 'tts', 'application/json', '/CreateSpeech', '',
            self._generate_payload(text_to_speak), self._region, self._host)
        if r.content.startswith(b'{'):
            raise PyvonaException('Error fetching voice: {}'.format(r.content))
        else:
            fp.write(r.content)

    def speak(self, text_to_speak):
        """Speak a given text
        """
        if not pygame_available:
            raise PyvonaException(
                "Pygame not installed. Please install to use speech.")

        with tempfile.SpooledTemporaryFile() as f:
            with self.use_ogg_codec():
                self.fetch_voice_fp(text_to_speak, f)
            f.seek(0)
            channel = pygame.mixer.Channel(5)
            sound = pygame.mixer.Sound(f)
            channel.play(sound)
            while channel.get_busy():
                pass

    def list_voices(self):
        """Returns all the possible voices
        """
        r = self._send_amazon_auth_packet_v4(
            'POST', 'tts', 'application/json', '/ListVoices', '', '',
            self._region, self._host)
        return r.content

    def _generate_payload(self, text_to_speak):
        return json.dumps({
            'Input': {
                'Data': text_to_speak
            },
            'OutputFormat': {
                'Codec': self.codec.upper()
            },
            'Parameters': {
                'Rate': self.speech_rate,
                'SentenceBreak': self.sentence_break,
                'ParagraphBreak': self.paragraph_break
            },
            'Voice': {
                'Name': self.voice_name
            }
        })

    def _send_amazon_auth_packet_v4(self, method, service, content_type,
                                    canonical_uri, canonical_querystring,
                                    request_parameters, region, host):
        """Send a packet to a given amazon server using Amazon's signature Version 4,
        Returns the resulting response object
        """
        # Create date for headers and the credential string
        t = datetime.datetime.utcnow()
        amazon_date = t.strftime(_amazon_date_format)
        date_stamp = t.strftime(_date_format)

        # Step 1: Create canonical request
        payload_hash = self._sha_hash(request_parameters)

        canonical_headers = 'content-type:{}\n'.format(content_type)
        canonical_headers += 'host:{}\n'.format(host)
        canonical_headers += 'x-amz-content-sha256:{}\n'.format(payload_hash)
        canonical_headers += 'x-amz-date:{}\n'.format(amazon_date)

        canonical_request = '\n'.join([
            method, canonical_uri, canonical_querystring, canonical_headers,
            self.signed_headers, payload_hash])

        # Step 2: Create the string to sign
        credential_scope = '{}/{}/{}/aws4_request'.format(
            date_stamp, region, service)
        string_to_sign = '\n'.join([
            self.algorithm, amazon_date, credential_scope,
            self._sha_hash(canonical_request)])

        # Step 3: Calculate the signature
        signing_key = self._get_signature_key(
            self.secret_key, date_stamp, region, service)
        signature = hmac.new(
            signing_key, string_to_sign.encode('utf-8'),
            hashlib.sha256).hexdigest()

        # Step 4: Create the signed packet
        endpoint = 'https://{}{}'.format(host, canonical_uri)
        authorization_header = '{} Credential={}/{}, ' +\
            'SignedHeaders={}, Signature={}'
        authorization_header = authorization_header.format(
            self.algorithm, self.access_key, credential_scope,
            self.signed_headers, signature)
        headers = {
            'Host': host,
            'Content-type': content_type,
            'X-Amz-Date': amazon_date,
            'Authorization': authorization_header,
            'x-amz-content-sha256': payload_hash,
            'Content-Length': len(request_parameters)
        }
        # Send the packet and return the response
        return requests.post(endpoint, data=request_parameters,
                             headers=headers)

    def _sha_hash(self, to_hash):
        return hashlib.sha256(to_hash.encode('utf-8')).hexdigest()

    def _sign(self, key, msg):
        return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()

    def _get_signature_key(self, key, date_stamp, region_name, service_name):
        k_date = self._sign(('AWS4{}'.format(key)).encode('utf-8'), date_stamp)
        k_region = self._sign(k_date, region_name)
        k_service = self._sign(k_region, service_name)
        k_signing = self._sign(k_service, 'aws4_request')
        return k_signing

    def __init__(self, access_key, secret_key):
        """Set initial voice object parameters
        """
        self.region = 'us-east'
        self.voice_name = 'Brian'
        self.access_key = access_key
        self.secret_key = secret_key
        self.speech_rate = 'medium'
        self.sentence_break = 400
        self.paragraph_break = 650
        if pygame_available:
            pygame.mixer.init()
<|endoftext|>


<commit_before>##############################################################################
#
#    Copyright (C) 2015 Comunitea Servicios Tecnológicos
#    $Carlos Lombardía Rodríguez <carlos@comunitea.com>$
#
#    This program is free software: you can redistribute it and/or modify
#    it under the terms of the GNU Affero General Public License as published
#    by the Free Software Foundation, either version 3 of the License, or
#    (at your option) any later version.
#
#    This program is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU Affero General Public License for more details.
#
#    You should have received a copy of the GNU Affero General Public License
#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
#
##############################################################################
from odoo import api, exceptions, fields, models, _

class StockPicking(models.Model):
    _inherit = "stock.picking"
    _order = "priority desc, date desc, id desc"

    internal_notes = fields.Text()
    commercial = fields.Many2one('res.users')
    move_location_id = fields.\
        Many2one('stock.location', related='move_lines.location_id',
                 string='Moves origin location', readonly=True, store=True)
    move_location_dest_id = fields.\
        Many2one('stock.location', related='move_lines.location_dest_id',
                 string='Moves destination location', readonly=True,
                 store=True)

    def action_done(self):
        lot_obj = self.env["stock.production.lot"]
        mov_line_obj = self.env['stock.move.line']
        for picking in self:
            for move in picking.move_lines:
                if move.product_id.tracking != 'none' and \
                        move.state == 'assigned' and not move.quantity_done \
                        and not move.lots_text:
                    for line in move.move_line_ids:
                        line.qty_done = line.product_uom_qty
                if move.product_id.tracking == 'none' and \
                        move.state == 'assigned' and \
                        not move.quantity_done and not picking.block_picking:
                    move.quantity_done = move.product_uom_qty
                elif move.lots_text:
                    txlots = move.lots_text.split(',')
                    if len(txlots) != len(move.move_line_ids):
                        quantity = len(txlots) - len(move.move_line_ids)
                        if quantity >= 0 and \
                                len(txlots) == move.product_uom_qty:
                            for i in range(0, int(quantity)):
                                mov_line_obj.create(
                                    move._prepare_move_line_vals())
                            move.refresh()
                        else:
                            raise exceptions.\
                                Warning(_("The number of lots defined"
                                          " are not equal to move"
                                          " product quantity"))
                    cont = 0
                    while (txlots):
                        lot_name = txlots.pop()
                        lot = lot_obj.search([("name", "=", lot_name),
                                              ("product_id", "=",
                                               move.product_id.id)],
                                             limit=1)
                        if not lot:
                            lot = lot_obj.create({'name': lot_name,
                                                  'product_id':
                                                  move.product_id.id})
                        move.move_line_ids[cont].\
                            write({'lot_id': lot.id,
                                   'qty_done': 1.0})
                        cont += 1
        res = super().action_done()
        for picking in self:
            picking_states = picking.sale_id.picking_ids.mapped('state')
            if all(state in ('done', 'cancel') for state in picking_states) \
                    and not all(state == 'cancel' for state in picking_states):
                picking.sale_id.action_done()
        return res

    @api.multi
    def write(self, vals):
        pickings_to_send = []
        for picking in self:
            # We do this huge condition to ensure that both fields are not empty when the mail is sent
            if ((vals.get('carrier_tracking_ref', False) and picking.carrier_name and not picking.carrier_tracking_ref) or
                    (vals.get('carrier_name', False) and picking.carrier_tracking_ref and not picking.carrier_name) or
                    (vals.get('carrier_name', False) and vals.get('carrier_tracking_ref', False) and not picking.carrier_name and not picking.carrier_tracking_ref)) and\
                    picking.picking_type_code == 'outgoing' and picking.sale_id:
                pickings_to_send.append(picking.id)
        result = super().write(vals)
        if pickings_to_send:
            pickings = self.env["stock.picking"].browse(pickings_to_send)
            for picking in pickings:
                # We need to do this after the write, otherwise the email template won't get well some  picking values
                picking_template = self.env.ref('stock_custom.picking_done_template')
                picking_template.with_context(lang=picking.partner_id.commercial_partner_id.lang).send_mail(picking.id)
        return result

class StockMoveLine(models.Model):

    _inherit = 'stock.move.line'

    sale_line = fields.Many2one('sale.order.line', store=True)
    sale_price_unit = fields.Float(store=True)
    sale_discount = fields.Float(store=True)
    sale_tax_description = fields.Char(store=True)
    sale_price_subtotal = fields.Monetary(store=True)
    sale_price_tax = fields.Float(store=True)
    sale_price_total = fields.Monetary(store=True)
    date_expected = fields.Datetime(related='move_id.date_expected',
                                    string="Date Expected")

    @api.depends('sale_line', 'sale_line.currency_id', 'sale_line.tax_id')
    def _compute_sale_order_line_fields(self):
        return super()._compute_sale_order_line_fields()

class StockMove(models.Model):
    _inherit = "stock.move"

    _order = 'date_expected asc, id'

    real_stock = fields.Float(compute='_compute_real_stock')
    available_stock = fields.Float(compute="_compute_available_stock")
    user_id = fields.Many2one('res.users', compute='_compute_responsible')
    lots_text = fields.Text('Lots', help="Value must be separated by commas")
    sale_id = fields.Many2one('sale.order', related='sale_line_id.order_id',
                              readonly=True)
    date_reliability = fields.Selection([
        ('1.high', 'High'),
        ('2.medium', 'Medium'),
        ('3.low', 'Low'),
        ], readonly=False,compute='_compute_dates')

    date_done = fields.Datetime(related='picking_id.date_done',store=True)

    def _compute_is_initial_demand_editable(self):
        super()._compute_is_initial_demand_editable()
        for move in self:
            if move.picking_id.state == 'draft' or (move.picking_id.state in ('confirmed', 'assigned') and move.picking_id.picking_type_id.code == 'incoming'):
                move.is_initial_demand_editable = True

    def _compute_responsible(self):
        for move in self:
            responsible = None
            if move.picking_id:
                responsible = move.picking_id.commercial.id
            elif move.origin:
                responsible = move.env['sale.order'].search(
                    [('name', '=', move.origin)]).user_id.id
            move.user_id = responsible

    def _compute_available_stock(self):
        for move in self:
            move.available_stock = move.product_id.virtual_stock_conservative

    def _compute_real_stock(self):
        for move in self:
            move.real_stock = move.product_id.qty_available

    def _compute_dates(self):
        for move in self:
            if move.picking_id:
                move.date_reliability="1.high"
            elif move.container_id:
                move.date_reliability="2.medium"
            else:
                move.date_reliability="3.low"

    @api.multi
    def _action_done(self):
        res = super()._action_done()
        stock_loc = self.env.ref("stock.stock_location_stock")
        for move in self:
            if (move.location_id.usage in ('supplier', 'production')) and \
                    (move.product_id.cost_method == 'fifo'):
                move.product_id.product_tmpl_id.recalculate_standard_price_2()
            if move.location_dest_id == stock_loc:
                domain = [('state', 'in', ['confirmed',
                                           'partially_available']),
                          ('picking_type_code', '=', 'outgoing'),
                          ('product_id', '=', move.product_id.id)]
                confirmed_ids = self.\
                    search(domain, limit=None,
                           order="has_reservations,sequence,picking_id,id")
                if confirmed_ids:
                    confirmed_ids._action_assign()
        return res

    def _get_price_unit(self):
        res = super()._get_price_unit()
        if not res:
            res = self.product_id.standard_price_2
        return res

    def action_do_unreserve(self):
        for move in self:
            if not move.mapped('move_line_ids'):
                move.state = 'confirmed'
        return self._do_unreserve()

    def action_force_assign(self):
        return self._force_assign()

    @api.multi
    def write(self,vals):
        res = super(StockMove, self).write(vals)
        for move in self:
            if move.purchase_line_id and move.product_id.date_first_incoming_reliability!='1.received' and (vals.get('date_expected') or vals.get('state') =='cancel' or vals.get('picking_id')==False):
                move.product_id._compute_date_first_incoming()
            if vals.get('date_expected') and move.purchase_line_id != False and move.state not in ['cancel','done'] and move.location_dest_id.usage=='internal':
                move.product_id.with_delay().update_product()
        return res

    @api.multi
    def create(self, vals):
        res = super(StockMove, self).create(vals)
        if vals.get('date_expected') and vals.get('purchase_line_id') and vals.get('state') not in ['cancel',
                                                                                'done'] and self.env['stock.location'].browse(vals.get('location_dest_id')).usage == 'internal':
                self.env['product.product'].browse(vals.get('product_id')).with_delay(eta=60).update_product()
        return res

class StockReturnPicking(models.TransientModel):
    _inherit = 'stock.return.picking'

    def _create_returns(self):
        new_picking, pick_type_id = super()._create_returns()
        pick_type_obj = self.env["stock.picking.type"].browse(pick_type_id)
        if pick_type_obj.code == "incoming":
            pick_obj = self.env["stock.picking"].browse(new_picking)
            for move in pick_obj.move_lines:
                if move.warehouse_id.lot_stock_id == move.location_dest_id:
                    move.location_dest_id = \
                        move.warehouse_id.wh_input_stock_loc_id.id
        return new_picking, pick_type_id

class StockReservation(models.Model):
    _inherit = 'stock.reservation'

    next_reception_date = fields.Date(compute='_compute_next_reception_date')

    def _compute_next_reception_date(self):
        for res in self:
            date_expected = False
            supp_id = self.env.ref('stock.stock_location_suppliers').id
            # First move search: Supplier to Playa
            moves = self.env['stock.move'].search(
                [('state', 'in', ('waiting', 'confirmed', 'assigned',
                                  'partially_available')),
                 ('product_id', '=', res.product_id.id),
                 ('location_id', '=', supp_id),
                 ('location_dest_id', '=',
                  res.sale_id.warehouse_id.wh_input_stock_loc_id.id)],
                order='date_expected asc')
            if not moves:
                customer_loc_id = self.env.ref('stock.stock_location_customers').id
                # Second move search: Customer to Playa
                moves = self.env['stock.move'].search(
                    [('state', 'in', ('waiting', 'confirmed', 'assigned',
                                      'partially_available')),
                     ('product_id', '=', res.product_id.id),
                     ('location_id', '=', customer_loc_id),
                     ('location_dest_id', '=',
                      res.sale_id.warehouse_id.wh_input_stock_loc_id.id)],
                    order='date_expected asc')
                if not moves:
                    # Third move search: Playa to VT child location
                    moves = self.env['stock.move'].search(
                        [('state', 'in', ('waiting', 'confirmed', 'assigned',
                                          'partially_available')),
                         ('product_id', '=', res.product_id.id),
                         ('location_id', '=',
                          res.sale_id.warehouse_id.wh_input_stock_loc_id.id),
                         ('location_dest_id', 'child_of',
                          [res.sale_id.warehouse_id.view_location_id.id])],
                        order='date_expected asc')
            if moves:
                date_expected = moves[0].date_expected
            res.next_reception_date = date_expected

class StockProductionLot(models.Model):
    _inherit = 'stock.production.lot'

    partner_id = fields.Many2one(
        'res.partner', string='Customer',
        compute='_compute_partner_id',
        help='The last customer in possession of the product')
    lot_notes = fields.Text('Notes')
    order_id= fields.Many2one('sale.order',string='Order',compute='_compute_partner_id')
    picking_id = fields.Many2one('stock.picking',string='Picking',compute='_compute_partner_id')
    def _compute_partner_id(self):
        pass
        for lot in self:
            move_line = self.env['stock.move.line'].search(
                [('lot_id', '=', lot.id)], order="id desc", limit=1)
            if move_line:
                lot.partner_id = \
                    move_line.picking_id.partner_id.commercial_partner_id
                lot.order_id=move_line.move_id.sale_line_id.order_id
                lot.picking_id =move_line.picking_id
            else:
                lot.partner_id = False

class StockLandedCost(models.Model):

    _inherit = 'stock.landed.cost'

    def button_validate(self):
        res = super().button_validate()
        valuation_lines = self.valuation_adjustment_lines.\
            filtered(lambda line: line.move_id)
        valuation_lines.mapped('move_id.product_id.product_tmpl_id').\
            recalculate_standard_price_2()
        return res
<commit_msg>[FIX] stock_custom: Cambio en el criterio de ordenación de los stock.move pendientes para asignar reservas, ya que antes se hacía a la inversa<commit_after>##############################################################################
#
#    Copyright (C) 2015 Comunitea Servicios Tecnológicos
#    $Carlos Lombardía Rodríguez <carlos@comunitea.com>$
#
#    This program is free software: you can redistribute it and/or modify
#    it under the terms of the GNU Affero General Public License as published
#    by the Free Software Foundation, either version 3 of the License, or
#    (at your option) any later version.
#
#    This program is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU Affero General Public License for more details.
#
#    You should have received a copy of the GNU Affero General Public License
#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
#
##############################################################################
from odoo import api, exceptions, fields, models, _

class StockPicking(models.Model):
    _inherit = "stock.picking"
    _order = "priority desc, date desc, id desc"

    internal_notes = fields.Text()
    commercial = fields.Many2one('res.users')
    move_location_id = fields.\
        Many2one('stock.location', related='move_lines.location_id',
                 string='Moves origin location', readonly=True, store=True)
    move_location_dest_id = fields.\
        Many2one('stock.location', related='move_lines.location_dest_id',
                 string='Moves destination location', readonly=True,
                 store=True)

    def action_done(self):
        lot_obj = self.env["stock.production.lot"]
        mov_line_obj = self.env['stock.move.line']
        for picking in self:
            for move in picking.move_lines:
                if move.product_id.tracking != 'none' and \
                        move.state == 'assigned' and not move.quantity_done \
                        and not move.lots_text:
                    for line in move.move_line_ids:
                        line.qty_done = line.product_uom_qty
                if move.product_id.tracking == 'none' and \
                        move.state == 'assigned' and \
                        not move.quantity_done and not picking.block_picking:
                    move.quantity_done = move.product_uom_qty
                elif move.lots_text:
                    txlots = move.lots_text.split(',')
                    if len(txlots) != len(move.move_line_ids):
                        quantity = len(txlots) - len(move.move_line_ids)
                        if quantity >= 0 and \
                                len(txlots) == move.product_uom_qty:
                            for i in range(0, int(quantity)):
                                mov_line_obj.create(
                                    move._prepare_move_line_vals())
                            move.refresh()
                        else:
                            raise exceptions.\
                                Warning(_("The number of lots defined"
                                          " are not equal to move"
                                          " product quantity"))
                    cont = 0
                    while (txlots):
                        lot_name = txlots.pop()
                        lot = lot_obj.search([("name", "=", lot_name),
                                              ("product_id", "=",
                                               move.product_id.id)],
                                             limit=1)
                        if not lot:
                            lot = lot_obj.create({'name': lot_name,
                                                  'product_id':
                                                  move.product_id.id})
                        move.move_line_ids[cont].\
                            write({'lot_id': lot.id,
                                   'qty_done': 1.0})
                        cont += 1
        res = super().action_done()
        for picking in self:
            picking_states = picking.sale_id.picking_ids.mapped('state')
            if all(state in ('done', 'cancel') for state in picking_states) \
                    and not all(state == 'cancel' for state in picking_states):
                picking.sale_id.action_done()
        return res

    @api.multi
    def write(self, vals):
        pickings_to_send = []
        for picking in self:
            # We do this huge condition to ensure that both fields are not empty when the mail is sent
            if ((vals.get('carrier_tracking_ref', False) and picking.carrier_name and not picking.carrier_tracking_ref) or
                    (vals.get('carrier_name', False) and picking.carrier_tracking_ref and not picking.carrier_name) or
                    (vals.get('carrier_name', False) and vals.get('carrier_tracking_ref', False) and not picking.carrier_name and not picking.carrier_tracking_ref)) and\
                    picking.picking_type_code == 'outgoing' and picking.sale_id:
                pickings_to_send.append(picking.id)
        result = super().write(vals)
        if pickings_to_send:
            pickings = self.env["stock.picking"].browse(pickings_to_send)
            for picking in pickings:
                # We need to do this after the write, otherwise the email template won't get well some  picking values
                picking_template = self.env.ref('stock_custom.picking_done_template')
                picking_template.with_context(lang=picking.partner_id.commercial_partner_id.lang).send_mail(picking.id)
        return result

class StockMoveLine(models.Model):

    _inherit = 'stock.move.line'

    sale_line = fields.Many2one('sale.order.line', store=True)
    sale_price_unit = fields.Float(store=True)
    sale_discount = fields.Float(store=True)
    sale_tax_description = fields.Char(store=True)
    sale_price_subtotal = fields.Monetary(store=True)
    sale_price_tax = fields.Float(store=True)
    sale_price_total = fields.Monetary(store=True)
    date_expected = fields.Datetime(related='move_id.date_expected',
                                    string="Date Expected")

    @api.depends('sale_line', 'sale_line.currency_id', 'sale_line.tax_id')
    def _compute_sale_order_line_fields(self):
        return super()._compute_sale_order_line_fields()

class StockMove(models.Model):
    _inherit = "stock.move"

    _order = 'date_expected asc, id'

    real_stock = fields.Float(compute='_compute_real_stock')
    available_stock = fields.Float(compute="_compute_available_stock")
    user_id = fields.Many2one('res.users', compute='_compute_responsible')
    lots_text = fields.Text('Lots', help="Value must be separated by commas")
    sale_id = fields.Many2one('sale.order', related='sale_line_id.order_id',
                              readonly=True)
    date_reliability = fields.Selection([
        ('1.high', 'High'),
        ('2.medium', 'Medium'),
        ('3.low', 'Low'),
        ], readonly=False,compute='_compute_dates')

    date_done = fields.Datetime(related='picking_id.date_done',store=True)

    def _compute_is_initial_demand_editable(self):
        super()._compute_is_initial_demand_editable()
        for move in self:
            if move.picking_id.state == 'draft' or (move.picking_id.state in ('confirmed', 'assigned') and move.picking_id.picking_type_id.code == 'incoming'):
                move.is_initial_demand_editable = True

    def _compute_responsible(self):
        for move in self:
            responsible = None
            if move.picking_id:
                responsible = move.picking_id.commercial.id
            elif move.origin:
                responsible = move.env['sale.order'].search(
                    [('name', '=', move.origin)]).user_id.id
            move.user_id = responsible

    def _compute_available_stock(self):
        for move in self:
            move.available_stock = move.product_id.virtual_stock_conservative

    def _compute_real_stock(self):
        for move in self:
            move.real_stock = move.product_id.qty_available

    def _compute_dates(self):
        for move in self:
            if move.picking_id:
                move.date_reliability="1.high"
            elif move.container_id:
                move.date_reliability="2.medium"
            else:
                move.date_reliability="3.low"

    @api.multi
    def _action_done(self):
        res = super()._action_done()
        stock_loc = self.env.ref("stock.stock_location_stock")
        for move in self:
            if (move.location_id.usage in ('supplier', 'production')) and \
                    (move.product_id.cost_method == 'fifo'):
                move.product_id.product_tmpl_id.recalculate_standard_price_2()
            if move.location_dest_id == stock_loc:
                domain = [('state', 'in', ['confirmed',
                                           'partially_available']),
                          ('picking_type_code', '=', 'outgoing'),
                          ('product_id', '=', move.product_id.id)]
                confirmed_ids = self.\
                    search(domain, limit=None,
                           order="has_reservations,sequence,date_expected,id")
                if confirmed_ids:
                    confirmed_ids._action_assign()
        return res

    def _get_price_unit(self):
        res = super()._get_price_unit()
        if not res:
            res = self.product_id.standard_price_2
        return res

    def action_do_unreserve(self):
        for move in self:
            if not move.mapped('move_line_ids'):
                move.state = 'confirmed'
        return self._do_unreserve()

    def action_force_assign(self):
        return self._force_assign()

    @api.multi
    def write(self,vals):
        res = super(StockMove, self).write(vals)
        for move in self:
            if move.purchase_line_id and move.product_id.date_first_incoming_reliability!='1.received' and (vals.get('date_expected') or vals.get('state') =='cancel' or vals.get('picking_id')==False):
                move.product_id._compute_date_first_incoming()
            if vals.get('date_expected') and move.purchase_line_id != False and move.state not in ['cancel','done'] and move.location_dest_id.usage=='internal':
                move.product_id.with_delay().update_product()
        return res

    @api.multi
    def create(self, vals):
        res = super(StockMove, self).create(vals)
        if vals.get('date_expected') and vals.get('purchase_line_id') and vals.get('state') not in ['cancel',
                                                                                'done'] and self.env['stock.location'].browse(vals.get('location_dest_id')).usage == 'internal':
                self.env['product.product'].browse(vals.get('product_id')).with_delay(eta=60).update_product()
        return res

class StockReturnPicking(models.TransientModel):
    _inherit = 'stock.return.picking'

    def _create_returns(self):
        new_picking, pick_type_id = super()._create_returns()
        pick_type_obj = self.env["stock.picking.type"].browse(pick_type_id)
        if pick_type_obj.code == "incoming":
            pick_obj = self.env["stock.picking"].browse(new_picking)
            for move in pick_obj.move_lines:
                if move.warehouse_id.lot_stock_id == move.location_dest_id:
                    move.location_dest_id = \
                        move.warehouse_id.wh_input_stock_loc_id.id
        return new_picking, pick_type_id

class StockReservation(models.Model):
    _inherit = 'stock.reservation'

    next_reception_date = fields.Date(compute='_compute_next_reception_date')

    def _compute_next_reception_date(self):
        for res in self:
            date_expected = False
            supp_id = self.env.ref('stock.stock_location_suppliers').id
            # First move search: Supplier to Playa
            moves = self.env['stock.move'].search(
                [('state', 'in', ('waiting', 'confirmed', 'assigned',
                                  'partially_available')),
                 ('product_id', '=', res.product_id.id),
                 ('location_id', '=', supp_id),
                 ('location_dest_id', '=',
                  res.sale_id.warehouse_id.wh_input_stock_loc_id.id)],
                order='date_expected asc')
            if not moves:
                customer_loc_id = self.env.ref('stock.stock_location_customers').id
                # Second move search: Customer to Playa
                moves = self.env['stock.move'].search(
                    [('state', 'in', ('waiting', 'confirmed', 'assigned',
                                      'partially_available')),
                     ('product_id', '=', res.product_id.id),
                     ('location_id', '=', customer_loc_id),
                     ('location_dest_id', '=',
                      res.sale_id.warehouse_id.wh_input_stock_loc_id.id)],
                    order='date_expected asc')
                if not moves:
                    # Third move search: Playa to VT child location
                    moves = self.env['stock.move'].search(
                        [('state', 'in', ('waiting', 'confirmed', 'assigned',
                                          'partially_available')),
                         ('product_id', '=', res.product_id.id),
                         ('location_id', '=',
                          res.sale_id.warehouse_id.wh_input_stock_loc_id.id),
                         ('location_dest_id', 'child_of',
                          [res.sale_id.warehouse_id.view_location_id.id])],
                        order='date_expected asc')
            if moves:
                date_expected = moves[0].date_expected
            res.next_reception_date = date_expected

class StockProductionLot(models.Model):
    _inherit = 'stock.production.lot'

    partner_id = fields.Many2one(
        'res.partner', string='Customer',
        compute='_compute_partner_id',
        help='The last customer in possession of the product')
    lot_notes = fields.Text('Notes')
    order_id= fields.Many2one('sale.order',string='Order',compute='_compute_partner_id')
    picking_id = fields.Many2one('stock.picking',string='Picking',compute='_compute_partner_id')
    def _compute_partner_id(self):
        pass
        for lot in self:
            move_line = self.env['stock.move.line'].search(
                [('lot_id', '=', lot.id)], order="id desc", limit=1)
            if move_line:
                lot.partner_id = \
                    move_line.picking_id.partner_id.commercial_partner_id
                lot.order_id=move_line.move_id.sale_line_id.order_id
                lot.picking_id =move_line.picking_id
            else:
                lot.partner_id = False

class StockLandedCost(models.Model):

    _inherit = 'stock.landed.cost'

    def button_validate(self):
        res = super().button_validate()
        valuation_lines = self.valuation_adjustment_lines.\
            filtered(lambda line: line.move_id)
        valuation_lines.mapped('move_id.product_id.product_tmpl_id').\
            recalculate_standard_price_2()
        return res
<|endoftext|>


<commit_before>import logging

from .baseflumine import BaseFlumine
from .events.events import EventType
from . import worker

logger = logging.getLogger(__name__)

class Flumine(BaseFlumine):
    def run(self) -> None:
        """
        Main run thread
        """
        with self:
            while True:
                event = self.handler_queue.get()
                if event.EVENT_TYPE == EventType.TERMINATOR:
                    self._process_end_flumine()
                    break

                elif event.EVENT_TYPE == EventType.MARKET_CATALOGUE:
                    self._process_market_catalogues(event)

                elif event.EVENT_TYPE == EventType.MARKET_BOOK:
                    self._process_market_books(event)

                elif event.EVENT_TYPE == EventType.RAW_DATA:
                    self._process_raw_data(event)

                elif event.EVENT_TYPE == EventType.CURRENT_ORDERS:
                    self._process_current_orders(event)

                elif event.EVENT_TYPE == EventType.ORDER_PACKAGE:
                    self._process_order_package(event)

                elif event.EVENT_TYPE == EventType.CLEARED_MARKETS:
                    self._process_cleared_markets(event)

                elif event.EVENT_TYPE == EventType.CLEARED_ORDERS:
                    self._process_cleared_orders(event)

                elif event.EVENT_TYPE == EventType.CLOSE_MARKET:
                    self._process_close_market(event)

                elif event.EVENT_TYPE == EventType.CUSTOM_EVENT:
                    self._process_custom_event(event)

                elif event.EVENT_TYPE == EventType.NEW_DAY:
                    logger.info(event)

                else:
                    logger.error("Unknown item in handler_queue: %s" % str(event))

    def _add_default_workers(self):
        self.add_worker(
            worker.BackgroundWorker(
                self,
                function=worker.keep_alive,
                interval=1200,
                name="keep_alive",
                func_args=(self.client,),
            )
        )
        self.add_worker(
            worker.BackgroundWorker(
                self,
                function=worker.poll_account_balance,
                interval=60,
                start_delay=5,  # wait for login
                name="poll_account_balance",
                func_args=(self, self.client),
            )
        )
        self.add_worker(
            worker.BackgroundWorker(
                self,
                function=worker.poll_market_catalogue,
                interval=60,
                start_delay=5,  # wait for streams to populate
                name="poll_market_catalogue",
                func_args=(self.client, self.markets, self.handler_queue),
            )
        )
        self.add_worker(
            worker.BackgroundWorker(
                self,
                function=worker.poll_cleared_orders,
                interval=10,  # restart
                name="poll_cleared_orders",
                func_args=(self, self.client),
            )
        )

    def __repr__(self) -> str:
        return "<Flumine>"

    def __str__(self) -> str:
        return "<Flumine>"
<commit_msg>bugfix<commit_after>import logging

from .baseflumine import BaseFlumine
from .events.events import EventType
from . import worker

logger = logging.getLogger(__name__)

class Flumine(BaseFlumine):
    def run(self) -> None:
        """
        Main run thread
        """
        with self:
            while True:
                event = self.handler_queue.get()
                if event.EVENT_TYPE == EventType.TERMINATOR:
                    self._process_end_flumine()
                    break

                elif event.EVENT_TYPE == EventType.MARKET_CATALOGUE:
                    self._process_market_catalogues(event)

                elif event.EVENT_TYPE == EventType.MARKET_BOOK:
                    self._process_market_books(event)

                elif event.EVENT_TYPE == EventType.RAW_DATA:
                    self._process_raw_data(event)

                elif event.EVENT_TYPE == EventType.CURRENT_ORDERS:
                    self._process_current_orders(event)

                elif event.EVENT_TYPE == EventType.ORDER_PACKAGE:
                    self._process_order_package(event)

                elif event.EVENT_TYPE == EventType.CLEARED_MARKETS:
                    self._process_cleared_markets(event)

                elif event.EVENT_TYPE == EventType.CLEARED_ORDERS:
                    self._process_cleared_orders(event)

                elif event.EVENT_TYPE == EventType.CLOSE_MARKET:
                    self._process_close_market(event)

                elif event.EVENT_TYPE == EventType.CUSTOM_EVENT:
                    self._process_custom_event(event)

                elif event.EVENT_TYPE == EventType.NEW_DAY:
                    logger.info(event)

                else:
                    logger.error("Unknown item in handler_queue: %s" % str(event))

    def _add_default_workers(self):
        self.add_worker(
            worker.BackgroundWorker(
                self,
                function=worker.keep_alive,
                interval=1200,
                name="keep_alive",
            )
        )
        self.add_worker(
            worker.BackgroundWorker(
                self,
                function=worker.poll_account_balance,
                interval=60,
                start_delay=5,  # wait for login
                name="poll_account_balance",
            )
        )
        self.add_worker(
            worker.BackgroundWorker(
                self,
                function=worker.poll_market_catalogue,
                interval=60,
                start_delay=5,  # wait for streams to populate
                name="poll_market_catalogue",
            )
        )
        self.add_worker(
            worker.BackgroundWorker(
                self,
                function=worker.poll_cleared_orders,
                interval=10,  # restart
                name="poll_cleared_orders",
            )
        )

    def __repr__(self) -> str:
        return "<Flumine>"

    def __str__(self) -> str:
        return "<Flumine>"
<|endoftext|>


<commit_before>import statsmodels.api as sm
import numpy as np

star98 = sm.datasets.star98.load_pandas().data

formula = 'SUCCESS ~ LOWINC + PERASIAN + PERBLACK + PERHISP + PCTCHRT '
formula += '+ PCTYRRND + PERMINTE*AVYRSEXP*AVSALK + PERSPENK*PTRATIO*PCTAF'

dta = star98[["NABOVE", "NBELOW", "LOWINC", "PERASIAN", "PERBLACK", "PERHISP",
              "PCTCHRT", "PCTYRRND", "PERMINTE", "AVYRSEXP", "AVSALK",
              "PERSPENK", "PTRATIO", "PCTAF"]]

endog = dta["NABOVE"]/(dta["NABOVE"] + dta.pop("NBELOW"))
del dta["NABOVE"]
dta["SUCCESS"] = endog

endog = dta.pop("SUCCESS")
exog = dta
mod = sm.GLM(endog, exog, formula=formula, family=sm.families.Binomial()).fit()
<commit_msg>ENH: Add example for injecting user transform<commit_after>import statsmodels.api as sm
import numpy as np

star98 = sm.datasets.star98.load_pandas().data

formula = 'SUCCESS ~ LOWINC + PERASIAN + PERBLACK + PERHISP + PCTCHRT '
formula += '+ PCTYRRND + PERMINTE*AVYRSEXP*AVSALK + PERSPENK*PTRATIO*PCTAF'

dta = star98[["NABOVE", "NBELOW", "LOWINC", "PERASIAN", "PERBLACK", "PERHISP",
              "PCTCHRT", "PCTYRRND", "PERMINTE", "AVYRSEXP", "AVSALK",
              "PERSPENK", "PTRATIO", "PCTAF"]]

endog = dta["NABOVE"]/(dta["NABOVE"] + dta.pop("NBELOW"))
del dta["NABOVE"]
dta["SUCCESS"] = endog

endog = dta.pop("SUCCESS")
exog = dta
mod = sm.GLM(endog, exog, formula=formula, family=sm.families.Binomial()).fit()

# try passing a formula object, using user-injected code

def double_it(x):
    return 2*x

# What is the correct entry point for this? Should users be able to inject
# code into default_env or similar? I don't see a way to do this yet using
# the approach I have been using, it should be an argument to Desc
from charlton.builtins import builtins
builtins['double_it'] = double_it

formula = 'SUCCESS ~ double_it(LOWINC) + PERASIAN + PERBLACK + PERHISP + '
formula += 'PCTCHRT '
formula += '+ PCTYRRND + PERMINTE*AVYRSEXP*AVSALK + PERSPENK*PTRATIO*PCTAF'
mod2 = sm.GLM(endog, exog, formula=formula, family=sm.families.Binomial()).fit()
<|endoftext|>