<commit_before>//! This module contains shareable mutable containers designed for enabling
//! and detecting race conditions in thread synchronization testing code.
//!
//! # Motivation
//!
//! The main purpose of a thread synchronization protocol is to ensure that
//! some operations which are not atomic in hardware, such as writing to two
//! unrelated memory locations, appear to occur as atomic transactions from the
//! point of view of other threads: at any point of time, either a given
//! operation appears to be done, or it appears not to have started.
//!
//! Testing a thread synchronization primitive involves showing that
//! inconsistent states (where an operation appears to be half-performed) have a
//! negligible probability of being exposed to the outside world in all expected
//! usage scenarios. It is typically done by showing that a given non-atomic
//! operation is properly encapsulated by the transactional semantics of the
//! synchronization protocol, and will never appear as half-done to observers.
//!
//! Non-atomic operations are easier said than done, however, when the set of
//! operations which can be atomic in hardware is larger than most people think
//! (at the time of writing, current Intel CPUs can access memory in blocks of
//! 128 bits, and current NVidia GPUs can do so in blocks of 1024 bits), not
//! well-defined by the architecture (and thus subjected to increase in future
//! hardware), and highly dependent on the compiler's optimization choices in a
//! high-level programming language such as Rust.
//!
//! Which is why I think we need some types whose operations are guaranteed to
//! be non-atomic under a set of reasonable assumptions, and which can easily be
//! observed to be in an inconsistent state.
//!
//! # Functionality
//!
//! This module provides the RaceCell type, which can hold a value of a
//! certain type T like a `Cell<T>` would, but is guaranteed **not** to be read
//! or written to in a single atomic operation, even if the corresponding type T
//! can be atomically read or written to in hardware.
//!
//! Furthermore, a RaceCell can detect scenarios where it is read at the same
//! time as it is being written (which constitutes a read-after-write race
//! condition) and report them to the reader thread.
//!
//! Such "controlled data races" can, in turn, be used to detect failures in
//! thread synchronization protocols, manifesting as inconsistent shared states
//! being exposed to the outside world.
//!
//! # Requirements on T
//!
//! In principle, any Clone + Eq type T whose equality operator and clone()
//! implementation behave well even if the inner data is in a inconsistent state
//! (e.g. filled with random bits) could be used. This is true of all primitive
//! integral types and aggregates thereof, for example.
//!
//! However, in practice, unsynchronized concurrent read/write access to
//! arbitrary data from multiple threads constitutes a data race, which is
//! undefined behaviour in Rust even when it occurs inside an UnsafeCell. This
//! is problematic because rustc's optimizer allows itself to transform code
//! which relies on undefined behaviour however it likes, leading to breakages
//! in release builds such as infinite loops appearing out of nowhere.
//!
//! For this reason, we currently only support some specific types T which have
//! atomic load and store operations, implemented as part of an atomic wrapper.
//! Note that although individual loads and stores to T are atomic, loads and
//! stores to RaceCell<T> are still guaranteed not to be atomic.

#![deny(missing_docs)]

use std::sync::atomic::{AtomicBool, AtomicIsize, AtomicPtr, AtomicUsize, Ordering};

/// Shareable mutable container for triggering and detecting write-after-read
/// data races in a well-controlled fashion.
#[derive(Debug)]
pub struct RaceCell<T>
where
    T: AtomicData,
{
    /// Two copies of a value of type T are made. One is stored on the stack...
    local_contents: T::AtomicWrapper,

    /// ...and one is stored on the heap, which in all popular OSs is too far
    /// away from the stack to allow any significant probability of the hardware
    /// writing both copies in a single atomic transactions.
    ///
    /// Of course, a malicious optimizer could still use hardware transactional
    /// memory or a software emulation thereof to achieve this effect, but there
    /// are no performance benefits in doing so, and in fact it will rather have
    /// an averse effect on performance, so a realistic optimizer won't do it.
    ///
    remote_version: Box<T::AtomicWrapper>,
}
//
impl<T> RaceCell<T>
where
    T: AtomicData,
{
    /// Create a new RaceCell with a certain initial content
    pub fn new(value: T) -> Self {
        RaceCell {
            local_contents: T::AtomicWrapper::new(value.clone()),
            remote_version: Box::new(T::AtomicWrapper::new(value)),
        }
    }

    /// Update the internal contents of the RaceCell in a non-atomic fashion
    pub fn set(&self, value: T) {
        self.local_contents.relaxed_store(value.clone());
        self.remote_version.relaxed_store(value);
    }

    /// Read the current contents of the RaceCell, detecting any data race
    /// caused by a concurrently occurring write along the way.
    pub fn get(&self) -> Racey<T> {
        let local_data = self.local_contents.relaxed_load();
        let remote_data = self.remote_version.relaxed_load();
        if local_data == remote_data {
            Racey::Consistent(local_data)
        } else {
            Racey::Inconsistent
        }
    }
}
//
impl<T> Clone for RaceCell<T>
where
    T: AtomicData,
{
    /// Making RaceCells cloneable allows putting them in concurrent containers
    fn clone(&self) -> Self {
        let local_copy = self.local_contents.relaxed_load();
        let remote_copy = self.remote_version.relaxed_load();
        RaceCell {
            local_contents: T::AtomicWrapper::new(local_copy),
            remote_version: Box::new(T::AtomicWrapper::new(remote_copy)),
        }
    }
}
//
impl<T> Default for RaceCell<T>
where
    T: AtomicData + Default,
{
    /// A RaceCell has a default value if the inner type has
    fn default() -> Self {
        Self::new(T::default())
    }
}

/// This is the result of a RaceCell read
#[derive(Debug, Eq, PartialEq)]
pub enum Racey<U: AtomicData> {
    /// The RaceCell was internally consistent, and its content was copied
    Consistent(U),

    /// The RaceCell was internally inconsistent: a data race has occurred
    Inconsistent,
}

/// Requirements on the data held by a RaceCell
pub trait AtomicData: Clone + Eq + Sized {
    /// Atomic wrapper type for this data implementing relaxed atomic load/store
    type AtomicWrapper: AtomicLoadStore<Content = Self>;
}
///
/// Atomic wrapper type for a certain kind of value
///
/// For the implementation of RaceCell not to be considered as undefined
/// behaviour and trashed by the Rust compiler in release mode, the underlying
/// type must be loaded and stored atomically. This requires synchronizing
/// accesses to it using things like the std::sync::atomic::AtomicXyz wrappers.
///
/// The only guarantee that we need is that loads and stores are atomic. We do
/// not need any other memory ordering guarantee. This is why we allow for
/// more wrapper type implementation freedom by not specifying memory orderings,
/// and internally relying only on relaxed ordering.
///
pub trait AtomicLoadStore: Sized {
    /// Type of data that is being wrapped
    type Content: AtomicData<AtomicWrapper = Self>;

    /// Create an atomic wrapper for a value of type U
    fn new(v: Self::Content) -> Self;

    /// Atomically load a value from the wrapper
    fn relaxed_load(&self) -> Self::Content;

    /// Atomically store a new value into the wrapper
    fn relaxed_store(&self, val: Self::Content);
}
///
/// This macro implements support for non-generic standard atomic types
///
macro_rules! impl_atomic_data {
    ($($data:ty => $wrapper:ty),*) => ($(
        impl AtomicData for $data {
            type AtomicWrapper = $wrapper;
        }

        impl AtomicLoadStore for $wrapper {
            type Content = $data;

            fn new(v: $data) -> $wrapper {
                <$wrapper>::new(v)
            }

            fn relaxed_load(&self) -> $data {
                <$wrapper>::load(self, Ordering::Relaxed)
            }

            fn relaxed_store(&self, val: $data) {
                <$wrapper>::store(self, val, Ordering::Relaxed)
            }
        }
    )*)
}
///
impl_atomic_data! { bool  => AtomicBool,
isize => AtomicIsize,
usize => AtomicUsize }
///
/// Atomic pointers are a bit special as they are generic, for now we will just
/// treat them as a special case.
///
impl<V> AtomicData for *mut V {
    type AtomicWrapper = AtomicPtr<V>;
}
///
impl<V> AtomicLoadStore for AtomicPtr<V> {
    type Content = *mut V;

    fn new(v: *mut V) -> AtomicPtr<V> {
        <AtomicPtr<V>>::new(v)
    }

    fn relaxed_load(&self) -> *mut V {
        <AtomicPtr<V>>::load(self, Ordering::Relaxed)
    }

    fn relaxed_store(&self, val: *mut V) {
        <AtomicPtr<V>>::store(self, val, Ordering::Relaxed)
    }
}

// FIXME: The astute reader will have noted that any data could be theoretically
//        put in a RaceCell by using a Mutex as the AtomicWrapper. However, this
//        will only be implemented once Rust has specialization, to avoid
//        pessimizing the common case where a primitive type is enough.

/// Here are some RaceCell tests
#[cfg(test)]
mod tests {
    use super::{AtomicLoadStore, RaceCell, Racey};
    use std::sync::{Arc, Mutex};

    /// A RaceCell should be created in a consistent and correct state
    #[test]
    fn initial_state() {
        let cell = RaceCell::new(true);
        assert_eq!(cell.local_contents.relaxed_load(), true);
        assert_eq!(cell.remote_version.relaxed_load(), true);
    }

    /// Reading a consistent RaceCell should work as expected
    #[test]
    fn consistent_read() {
        let cell = RaceCell::new(-42_isize);
        assert_eq!(cell.get(), Racey::Consistent(-42));
    }

    /// Reading an inconsistent RaceCell should work as expected
    #[test]
    fn inconsistent_read() {
        let cell = RaceCell::new(0xbad_usize);
        cell.local_contents.relaxed_store(0xdead);
        assert_eq!(cell.get(), Racey::Inconsistent);
    }

    /// RaceCells should be cloned as-is, even if in an inconsistent state
    #[test]
    fn clone() {
        let cell = RaceCell::new(0xbeef_usize);
        cell.local_contents.relaxed_store(0xdeaf);
        let clone = cell.clone();
        assert_eq!(clone.local_contents.relaxed_load(), 0xdeaf);
        assert_eq!(clone.remote_version.relaxed_load(), 0xbeef);
    }

    /// Unprotected concurrent reads and writes to a RaceCell should trigger
    /// detectable race conditions, illustrating its non-atomic nature.
    ///
    /// To maximize the odds of race conditions, this kind of test should be run
    /// in single-threaded mode.
    ///
    #[test]
    #[ignore]
    fn unprotected_race() {
        // Amount of writes to carry out
        const WRITES_COUNT: usize = 100_000_000;

        // RaceCell in which the writes will be carried out
        let initial_value = 0usize;
        let cell1 = Arc::new(RaceCell::new(initial_value));
        let cell2 = cell1.clone();

        // Make sure that RaceCell does expose existing data races, with a
        // detection probability better than 1% for very obvious ones :)
        crate::concurrent_test_2(
            move || {
                for i in 1..(WRITES_COUNT + 1) {
                    cell1.set(i);
                }
            },
            move || {
                let mut last_value = 0;
                let mut data_race_count = 0usize;
                while last_value != WRITES_COUNT {
                    match cell2.get() {
                        Racey::Consistent(value) => last_value = value,
                        Racey::Inconsistent => data_race_count += 1,
                    }
                }
                print!("{} races detected: ", data_race_count);
                assert!(data_race_count > WRITES_COUNT / 100);
            },
        );
    }

    /// Appropriately protected concurrent reads and writes to a RaceCell should
    /// not yield any detectable race conditions.
    ///
    /// To maximize the odds of race conditions, this kind of test should be run
    /// in single-threaded mode.
    ///
    #[test]
    #[ignore]
    fn protected_transaction() {
        // Amount of writes to carry out
        const WRITES_COUNT: usize = 10_000_000;

        // Mutex-protected RaceCell in which the writes will be carried out
        let initial_value = 0usize;
        let cell1 = Arc::new(Mutex::new(RaceCell::new(initial_value)));
        let cell2 = cell1.clone();

        // Make sure that RaceCell does not incorrectly detect race conditions
        crate::concurrent_test_2(
            move || {
                for i in 1..(WRITES_COUNT + 1) {
                    cell1.lock().unwrap().set(i);
                }
            },
            move || {
                let mut last_value = 0;
                let mut data_race_count = 0usize;
                while last_value != WRITES_COUNT {
                    match cell2.lock().unwrap().get() {
                        Racey::Consistent(value) => last_value = value,
                        Racey::Inconsistent => data_race_count += 1,
                    }
                }
                assert_eq!(data_race_count, 0);
            },
        );
    }
}
<commit_msg>A bit of manual cleanup<commit_after>//! This module contains shareable mutable containers designed for enabling
//! and detecting race conditions in thread synchronization testing code.
//!
//! # Motivation
//!
//! The main purpose of a thread synchronization protocol is to ensure that
//! some operations which are not atomic in hardware, such as writing to two
//! unrelated memory locations, appear to occur as atomic transactions from the
//! point of view of other threads: at any point of time, either a given
//! operation appears to be done, or it appears not to have started.
//!
//! Testing a thread synchronization primitive involves showing that
//! inconsistent states (where an operation appears to be half-performed) have a
//! negligible probability of being exposed to the outside world in all expected
//! usage scenarios. It is typically done by showing that a given non-atomic
//! operation is properly encapsulated by the transactional semantics of the
//! synchronization protocol, and will never appear as half-done to observers.
//!
//! Non-atomic operations are easier said than done, however, when the set of
//! operations which can be atomic in hardware is larger than most people think
//! (at the time of writing, current Intel CPUs can access memory in blocks of
//! 128 bits, and current NVidia GPUs can do so in blocks of 1024 bits), not
//! well-defined by the architecture (and thus subjected to increase in future
//! hardware), and highly dependent on the compiler's optimization choices in a
//! high-level programming language such as Rust.
//!
//! Which is why I think we need some types whose operations are guaranteed to
//! be non-atomic under a set of reasonable assumptions, and which can easily be
//! observed to be in an inconsistent state.
//!
//! # Functionality
//!
//! This module provides the RaceCell type, which can hold a value of a
//! certain type T like a `Cell<T>` would, but is guaranteed **not** to be read
//! or written to in a single atomic operation, even if the corresponding type T
//! can be atomically read or written to in hardware.
//!
//! Furthermore, a RaceCell can detect scenarios where it is read at the same
//! time as it is being written (which constitutes a read-after-write race
//! condition) and report them to the reader thread.
//!
//! Such "controlled data races" can, in turn, be used to detect failures in
//! thread synchronization protocols, manifesting as inconsistent shared states
//! being exposed to the outside world.
//!
//! # Requirements on T
//!
//! In principle, any Clone + Eq type T whose equality operator and clone()
//! implementation behave well even if the inner data is in a inconsistent state
//! (e.g. filled with random bits) could be used. This is true of all primitive
//! integral types and aggregates thereof, for example.
//!
//! However, in practice, unsynchronized concurrent read/write access to
//! arbitrary data from multiple threads constitutes a data race, which is
//! undefined behaviour in Rust even when it occurs inside an UnsafeCell. This
//! is problematic because rustc's optimizer allows itself to transform code
//! which relies on undefined behaviour however it likes, leading to breakages
//! in release builds such as infinite loops appearing out of nowhere.
//!
//! For this reason, we currently only support some specific types T which have
//! atomic load and store operations, implemented as part of an atomic wrapper.
//! Note that although individual loads and stores to T are atomic, loads and
//! stores to RaceCell<T> are still guaranteed not to be atomic.

#![deny(missing_docs)]

use std::sync::atomic::{AtomicBool, AtomicIsize, AtomicPtr, AtomicUsize, Ordering};

/// Shareable mutable container for triggering and detecting write-after-read
/// data races in a well-controlled fashion.
#[derive(Debug)]
pub struct RaceCell<T: AtomicData> {
    /// Two copies of a value of type T are made. One is stored on the stack...
    local_contents: T::AtomicWrapper,

    /// ...and one is stored on the heap, which in all popular OSs is too far
    /// away from the stack to allow any significant probability of the hardware
    /// writing both copies in a single atomic transactions.
    ///
    /// Of course, a malicious optimizer could still use hardware transactional
    /// memory or a software emulation thereof to achieve this effect, but there
    /// are no performance benefits in doing so, and in fact it will rather have
    /// an averse effect on performance, so a realistic optimizer won't do it.
    ///
    remote_version: Box<T::AtomicWrapper>,
}
//
impl<T: AtomicData> RaceCell<T> {
    /// Create a new RaceCell with a certain initial content
    pub fn new(value: T) -> Self {
        RaceCell {
            local_contents: T::AtomicWrapper::new(value.clone()),
            remote_version: Box::new(T::AtomicWrapper::new(value)),
        }
    }

    /// Update the internal contents of the RaceCell in a non-atomic fashion
    pub fn set(&self, value: T) {
        self.local_contents.relaxed_store(value.clone());
        self.remote_version.relaxed_store(value);
    }

    /// Read the current contents of the RaceCell, detecting any data race
    /// caused by a concurrently occurring write along the way.
    pub fn get(&self) -> Racey<T> {
        let local_data = self.local_contents.relaxed_load();
        let remote_data = self.remote_version.relaxed_load();
        if local_data == remote_data {
            Racey::Consistent(local_data)
        } else {
            Racey::Inconsistent
        }
    }
}
//
impl<T: AtomicData> Clone for RaceCell<T> {
    /// Making RaceCells cloneable allows putting them in concurrent containers
    fn clone(&self) -> Self {
        let local_copy = self.local_contents.relaxed_load();
        let remote_copy = self.remote_version.relaxed_load();
        RaceCell {
            local_contents: T::AtomicWrapper::new(local_copy),
            remote_version: Box::new(T::AtomicWrapper::new(remote_copy)),
        }
    }
}
//
impl<T: AtomicData + Default> Default for RaceCell<T> {
    /// A RaceCell has a default value if the inner type has
    fn default() -> Self {
        Self::new(T::default())
    }
}

/// This is the result of a RaceCell read
#[derive(Debug, Eq, PartialEq)]
pub enum Racey<U: AtomicData> {
    /// The RaceCell was internally consistent, and its content was copied
    Consistent(U),

    /// The RaceCell was internally inconsistent: a data race has occurred
    Inconsistent,
}

/// Requirements on the data held by a RaceCell
pub trait AtomicData: Clone + Eq + Sized {
    /// Atomic wrapper type for this data implementing relaxed atomic load/store
    type AtomicWrapper: AtomicLoadStore<Content = Self>;
}
///
/// Atomic wrapper type for a certain kind of value
///
/// For the implementation of RaceCell not to be considered as undefined
/// behaviour and trashed by the Rust compiler in release mode, the underlying
/// type must be loaded and stored atomically. This requires synchronizing
/// accesses to it using things like the std::sync::atomic::AtomicXyz wrappers.
///
/// The only guarantee that we need is that loads and stores are atomic. We do
/// not need any other memory ordering guarantee. This is why we allow for
/// more wrapper type implementation freedom by not specifying memory orderings,
/// and internally relying only on relaxed ordering.
///
pub trait AtomicLoadStore: Sized {
    /// Type of data that is being wrapped
    type Content: AtomicData<AtomicWrapper = Self>;

    /// Create an atomic wrapper for a value of type U
    fn new(v: Self::Content) -> Self;

    /// Atomically load a value from the wrapper
    fn relaxed_load(&self) -> Self::Content;

    /// Atomically store a new value into the wrapper
    fn relaxed_store(&self, val: Self::Content);
}
///
/// This macro implements support for non-generic standard atomic types
///
macro_rules! impl_atomic_data {
    ($($data:ty => $wrapper:ty),*) => ($(
        impl AtomicData for $data {
            type AtomicWrapper = $wrapper;
        }

        impl AtomicLoadStore for $wrapper {
            type Content = $data;

            fn new(v: $data) -> $wrapper {
                <$wrapper>::new(v)
            }

            fn relaxed_load(&self) -> $data {
                <$wrapper>::load(self, Ordering::Relaxed)
            }

            fn relaxed_store(&self, val: $data) {
                <$wrapper>::store(self, val, Ordering::Relaxed)
            }
        }
    )*)
}
///
impl_atomic_data! {
    bool  => AtomicBool,
    isize => AtomicIsize,
    usize => AtomicUsize
}
///
/// Atomic pointers are a bit special as they are generic, for now we will just
/// treat them as a special case.
///
impl<V> AtomicData for *mut V {
    type AtomicWrapper = AtomicPtr<V>;
}
///
impl<V> AtomicLoadStore for AtomicPtr<V> {
    type Content = *mut V;

    fn new(v: *mut V) -> AtomicPtr<V> {
        <AtomicPtr<V>>::new(v)
    }

    fn relaxed_load(&self) -> *mut V {
        <AtomicPtr<V>>::load(self, Ordering::Relaxed)
    }

    fn relaxed_store(&self, val: *mut V) {
        <AtomicPtr<V>>::store(self, val, Ordering::Relaxed)
    }
}

// FIXME: The astute reader will have noted that any data could be theoretically
//        put in a RaceCell by using a Mutex as the AtomicWrapper. However, this
//        will only be implemented once Rust has specialization, to avoid
//        pessimizing the common case where a primitive type is enough.

/// Here are some RaceCell tests
#[cfg(test)]
mod tests {
    use super::{AtomicLoadStore, RaceCell, Racey};
    use std::sync::{Arc, Mutex};

    /// A RaceCell should be created in a consistent and correct state
    #[test]
    fn initial_state() {
        let cell = RaceCell::new(true);
        assert_eq!(cell.local_contents.relaxed_load(), true);
        assert_eq!(cell.remote_version.relaxed_load(), true);
    }

    /// Reading a consistent RaceCell should work as expected
    #[test]
    fn consistent_read() {
        let cell = RaceCell::new(-42_isize);
        assert_eq!(cell.get(), Racey::Consistent(-42));
    }

    /// Reading an inconsistent RaceCell should work as expected
    #[test]
    fn inconsistent_read() {
        let cell = RaceCell::new(0xbad_usize);
        cell.local_contents.relaxed_store(0xdead);
        assert_eq!(cell.get(), Racey::Inconsistent);
    }

    /// RaceCells should be cloned as-is, even if in an inconsistent state
    #[test]
    fn clone() {
        let cell = RaceCell::new(0xbeef_usize);
        cell.local_contents.relaxed_store(0xdeaf);
        let clone = cell.clone();
        assert_eq!(clone.local_contents.relaxed_load(), 0xdeaf);
        assert_eq!(clone.remote_version.relaxed_load(), 0xbeef);
    }

    /// Unprotected concurrent reads and writes to a RaceCell should trigger
    /// detectable race conditions, illustrating its non-atomic nature.
    ///
    /// To maximize the odds of race conditions, this kind of test should be run
    /// in single-threaded mode.
    ///
    #[test]
    #[ignore]
    fn unprotected_race() {
        // Amount of writes to carry out
        const WRITES_COUNT: usize = 100_000_000;

        // RaceCell in which the writes will be carried out
        let initial_value = 0usize;
        let cell1 = Arc::new(RaceCell::new(initial_value));
        let cell2 = cell1.clone();

        // Make sure that RaceCell does expose existing data races, with a
        // detection probability better than 1% for very obvious ones :)
        crate::concurrent_test_2(
            move || {
                for i in 1..(WRITES_COUNT + 1) {
                    cell1.set(i);
                }
            },
            move || {
                let mut last_value = 0;
                let mut data_race_count = 0usize;
                while last_value != WRITES_COUNT {
                    match cell2.get() {
                        Racey::Consistent(value) => last_value = value,
                        Racey::Inconsistent => data_race_count += 1,
                    }
                }
                print!("{} races detected: ", data_race_count);
                assert!(data_race_count > WRITES_COUNT / 100);
            },
        );
    }

    /// Appropriately protected concurrent reads and writes to a RaceCell should
    /// not yield any detectable race conditions.
    ///
    /// To maximize the odds of race conditions, this kind of test should be run
    /// in single-threaded mode.
    ///
    #[test]
    #[ignore]
    fn protected_transaction() {
        // Amount of writes to carry out
        const WRITES_COUNT: usize = 10_000_000;

        // Mutex-protected RaceCell in which the writes will be carried out
        let initial_value = 0usize;
        let cell1 = Arc::new(Mutex::new(RaceCell::new(initial_value)));
        let cell2 = cell1.clone();

        // Make sure that RaceCell does not incorrectly detect race conditions
        crate::concurrent_test_2(
            move || {
                for i in 1..(WRITES_COUNT + 1) {
                    cell1.lock().unwrap().set(i);
                }
            },
            move || {
                let mut last_value = 0;
                let mut data_race_count = 0usize;
                while last_value != WRITES_COUNT {
                    match cell2.lock().unwrap().get() {
                        Racey::Consistent(value) => last_value = value,
                        Racey::Inconsistent => data_race_count += 1,
                    }
                }
                assert_eq!(data_race_count, 0);
            },
        );
    }
}
<|endoftext|>


<commit_before>use sass::output_style::SassOutputStyle;
use ast::node::Node;
use token::{Lexeme, Token};
use error::Result;
use context::Context;

use std::fmt;
use std::io::Write;

#[derive(Clone, PartialEq)]
pub struct SassRule {
    pub selectors: Vec<Lexeme>,
    pub children: Vec<Node>,
}

impl SassRule {
    pub fn new() -> SassRule {
        SassRule {
            selectors: Vec::new(),
            children: Vec::new(),
        }
    }

    fn selector_distribution(&self, parents: &str, separator: &str) -> String {
        match parents.len() {
            0 => self.selectors.iter().map(|s| s.token.to_string()).collect::<Vec<_>>().join(separator),
            _ => parents.split(",").map(|p| {
                self.selectors.iter().map(|s| {
                    if s.token.to_string().contains("&") {
                        s.token.to_string().replace("&", p.trim())
                    } else {
                        format!("{} {}", p.trim(), s.token)
                    }
                }).collect::<Vec<_>>().join(separator)
            }).collect::<Vec<_>>().join(separator),
        }
    }

    pub fn stream<W: Write>(&self, output: &mut W, style: SassOutputStyle) -> Result<()> {
        try!(self.recursive_stream(output, style, "", ""));
        Ok(try!(write!(output, "{}", style.rule_separator())))
    }

    pub fn recursive_stream<W: Write>(&self, output: &mut W, style: SassOutputStyle, parents: &str, nesting: &str) -> Result<()> {
        let mut selector_string = self.selector_distribution(parents, &style.selector_separator());
        if style == SassOutputStyle::Compressed {
            selector_string = compress_selectors(selector_string);
        }

        let mut properties = self.child_properties().into_iter();
        // let mut has_properties = false;

        // TODO: peek?
        if let Some(prop) = properties.next() {
            // has_properties = true;
            try!(write!(output, "{}{}{{{}{}",
              selector_string,
              style.selector_brace_separator(),
              style.brace_property_separator(),
              style.before_property(nesting),
            ));

            try!(prop.stream(output, style));

            for prop in properties {
                try!(write!(output, "{}{}",
                    style.after_property(),
                    style.before_property(nesting),
                ));
                try!(prop.stream(output, style));
            }
            try!(write!(output, "{}}}", style.property_brace_separator()));
        }

        Ok(())
    }

    // TODO: I don't like how child_properties and child_rules are so different, but
    // i'm not sure what to do about it yet.
    pub fn child_properties(&self) -> Vec<&Node> {
        self.children.iter().filter(|c|
            match **c {
                Node::Rule(..) => false,
                Node::Property(..) => true,
            }
        ).collect::<Vec<_>>()
    }

    pub fn child_rules(&self) -> Vec<SassRule> {
        self.children.clone().into_iter().filter_map(|c|
            match c {
                Node::Rule(rule) => Some(rule),
                Node::Property(..) => None,
            }
        ).collect::<Vec<_>>()
    }

    pub fn has_properties(&self) -> bool {
        !self.child_properties().is_empty()
    }

    pub fn optimize(self) -> Vec<SassRule> {
        let mut results = vec![];
        if self.has_properties() {
            results.push(self.clone());
        }
        for cr in self.child_rules().into_iter() {
            results.extend(cr.collapse_with_parent_selectors(&self.selectors));
        }
        results
    }

    pub fn collapse_with_parent_selectors(self, parents: &Vec<Lexeme>) -> Vec<SassRule> {
        let new_selectors = parents.iter().flat_map(|p|
            self.selectors.iter().map(|c|
                Lexeme {
                    token: Token::String(format!("{} {}", p.token, c.token)),
                    offset: p.offset,
                }
            ).collect::<Vec<_>>()
        ).collect();
        SassRule {
            selectors: new_selectors,
            children: self.children,
        }.optimize()
    }

    pub fn evaluate(self, context: &Context) -> SassRule {
        let local_context = context.clone();
        SassRule {
            selectors: self.selectors,
            children: self.children.into_iter().map(|c|
                match c {
                    Node::Rule(sr) => Node::Rule(sr.evaluate(&local_context)),
                    Node::Property(lex, ex) => Node::Property(lex, ex.evaluate(&local_context)),
                }
            ).collect(),
        }
    }
}

fn compress_selectors(selector_string: String) -> String {
    selector_string.replace(" > ", ">").replace(" + ", "+")
}

impl fmt::Debug for SassRule {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        let children = self.children.iter().map(|c| format!("{:?}", c)).collect::<Vec<_>>().join("\n");
        let indented_children = children.split("\n").collect::<Vec<_>>().join("\n  ");
        write!(f, "\n{:?} {{\n  {}\n}}", self.selectors, indented_children)
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use ast::node::Node;
    use ast::expression::Expression;
    use token::{Token, Lexeme};

    #[test]
    fn it_collapses_subrules_without_properties() {
        let innermost_rule = SassRule {
            selectors: vec![Lexeme {
                token: Token::String("strong".into()),
                offset: Some(6)
            }],
            children: vec![
                Node::Property(
                    Lexeme { token: Token::String("font-weight".into()), offset: Some(12) },
                    Expression::String(
                        Lexeme { token: Token::String("bold".into()), offset: Some(19) }
                    ),
                ),
            ],
        };

        let further_in_rule = SassRule {
            selectors: vec![Lexeme {
                token: Token::String("img".into()),
                offset: Some(6)
            }],
            children: vec![
                Node::Property(
                    Lexeme { token: Token::String("color".into()), offset: Some(12) },
                    Expression::String(
                        Lexeme { token: Token::String("blue".into()), offset: Some(19) }
                    ),
                ),
                Node::Rule(innermost_rule),
            ],
        };

        let middle_rule = SassRule {
            selectors: vec![Lexeme { token: Token::String("span".into()), offset: Some(0) }],
            children: vec![Node::Rule(further_in_rule)],
        };

        let outer_rule = SassRule {
            selectors: vec![Lexeme { token: Token::String("div".into()), offset: Some(0) }],
            children: vec![Node::Rule(middle_rule)],
        };

        assert_eq!(
            outer_rule.optimize(),
            vec![
                SassRule {
                    selectors: vec![
                        Lexeme { token: Token::String("div span img".into()), offset: Some(0) },
                    ],
                    children: vec![
                        Node::Property(
                            Lexeme { token: Token::String("color".into()), offset: Some(12) },
                            Expression::String(
                                Lexeme { token: Token::String("blue".into()), offset: Some(19) }
                            ),
                        ),
                        Node::Rule(
                            SassRule {
                                selectors: vec![
                                    Lexeme {
                                        token: Token::String("strong".into()),
                                        offset: Some(6)
                                    },
                                ],
                                children: vec![Node::Property(
                                    Lexeme { token: Token::String("font-weight".into()), offset: Some(12) },
                                    Expression::String(
                                        Lexeme { token: Token::String("bold".into()), offset: Some(19) }
                                    ),
                                )],
                            }
                        ),
                    ],
                },
            ]
        );
    }
}
<commit_msg>Collapsed child rules should remain child rules<commit_after>use sass::output_style::SassOutputStyle;
use ast::node::Node;
use token::{Lexeme, Token};
use error::Result;
use context::Context;

use std::fmt;
use std::io::Write;

#[derive(Clone, PartialEq)]
pub struct SassRule {
    pub selectors: Vec<Lexeme>,
    pub children: Vec<Node>,
}

impl SassRule {
    pub fn new() -> SassRule {
        SassRule {
            selectors: Vec::new(),
            children: Vec::new(),
        }
    }

    fn selector_distribution(&self, parents: &str, separator: &str) -> String {
        match parents.len() {
            0 => self.selectors.iter().map(|s| s.token.to_string()).collect::<Vec<_>>().join(separator),
            _ => parents.split(",").map(|p| {
                self.selectors.iter().map(|s| {
                    if s.token.to_string().contains("&") {
                        s.token.to_string().replace("&", p.trim())
                    } else {
                        format!("{} {}", p.trim(), s.token)
                    }
                }).collect::<Vec<_>>().join(separator)
            }).collect::<Vec<_>>().join(separator),
        }
    }

    pub fn stream<W: Write>(&self, output: &mut W, style: SassOutputStyle) -> Result<()> {
        try!(self.recursive_stream(output, style, "", ""));
        Ok(try!(write!(output, "{}", style.rule_separator())))
    }

    pub fn recursive_stream<W: Write>(&self, output: &mut W, style: SassOutputStyle, parents: &str, nesting: &str) -> Result<()> {
        let mut selector_string = self.selector_distribution(parents, &style.selector_separator());
        if style == SassOutputStyle::Compressed {
            selector_string = compress_selectors(selector_string);
        }

        let mut properties = self.child_properties().into_iter();
        // let mut has_properties = false;

        // TODO: peek?
        if let Some(prop) = properties.next() {
            // has_properties = true;
            try!(write!(output, "{}{}{{{}{}",
              selector_string,
              style.selector_brace_separator(),
              style.brace_property_separator(),
              style.before_property(nesting),
            ));

            try!(prop.stream(output, style));

            for prop in properties {
                try!(write!(output, "{}{}",
                    style.after_property(),
                    style.before_property(nesting),
                ));
                try!(prop.stream(output, style));
            }
            try!(write!(output, "{}}}", style.property_brace_separator()));
        }

        Ok(())
    }

    // TODO: I don't like how child_properties and child_rules are so different, but
    // i'm not sure what to do about it yet.
    pub fn child_properties(&self) -> Vec<&Node> {
        self.children.iter().filter(|c|
            match **c {
                Node::Rule(..) => false,
                Node::Property(..) => true,
            }
        ).collect::<Vec<_>>()
    }

    pub fn child_rules(&self) -> Vec<SassRule> {
        self.children.clone().into_iter().filter_map(|c|
            match c {
                Node::Rule(rule) => Some(rule),
                Node::Property(..) => None,
            }
        ).collect::<Vec<_>>()
    }

    pub fn has_properties(&self) -> bool {
        !self.child_properties().is_empty()
    }

    pub fn optimize(self) -> Vec<SassRule> {
        let mut results = vec![];
        if self.has_properties() {
            results.push(self.clone());
            return results
        } else {
            self.child_rules().into_iter().flat_map(|cr|
                cr.collapse_with_parent_selectors(&self.selectors)
            ).collect::<Vec<_>>()
        }
    }

    pub fn collapse_with_parent_selectors(self, parents: &Vec<Lexeme>) -> Vec<SassRule> {
        let new_selectors = parents.iter().flat_map(|p|
            self.selectors.iter().map(|c|
                Lexeme {
                    token: Token::String(format!("{} {}", p.token, c.token)),
                    offset: p.offset,
                }
            ).collect::<Vec<_>>()
        ).collect();
        SassRule {
            selectors: new_selectors,
            children: self.children,
        }.optimize()
    }

    pub fn evaluate(self, context: &Context) -> SassRule {
        let local_context = context.clone();
        SassRule {
            selectors: self.selectors,
            children: self.children.into_iter().map(|c|
                match c {
                    Node::Rule(sr) => Node::Rule(sr.evaluate(&local_context)),
                    Node::Property(lex, ex) => Node::Property(lex, ex.evaluate(&local_context)),
                }
            ).collect(),
        }
    }
}

fn compress_selectors(selector_string: String) -> String {
    selector_string.replace(" > ", ">").replace(" + ", "+")
}

impl fmt::Debug for SassRule {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        let children = self.children.iter().map(|c| format!("{:?}", c)).collect::<Vec<_>>().join("\n");
        let indented_children = children.split("\n").collect::<Vec<_>>().join("\n  ");
        write!(f, "\n{:?} {{\n  {}\n}}", self.selectors, indented_children)
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use ast::node::Node;
    use ast::expression::Expression;
    use token::{Token, Lexeme};

    #[test]
    fn it_collapses_subrules_without_properties() {
        let innermost_rule = SassRule {
            selectors: vec![Lexeme {
                token: Token::String("strong".into()),
                offset: Some(6)
            }],
            children: vec![
                Node::Property(
                    Lexeme { token: Token::String("font-weight".into()), offset: Some(12) },
                    Expression::String(
                        Lexeme { token: Token::String("bold".into()), offset: Some(19) }
                    ),
                ),
            ],
        };

        let further_in_rule = SassRule {
            selectors: vec![Lexeme {
                token: Token::String("img".into()),
                offset: Some(6)
            }],
            children: vec![
                Node::Property(
                    Lexeme { token: Token::String("color".into()), offset: Some(12) },
                    Expression::String(
                        Lexeme { token: Token::String("blue".into()), offset: Some(19) }
                    ),
                ),
                Node::Rule(innermost_rule),
            ],
        };

        let middle_rule = SassRule {
            selectors: vec![Lexeme { token: Token::String("span".into()), offset: Some(0) }],
            children: vec![Node::Rule(further_in_rule)],
        };

        let outer_rule = SassRule {
            selectors: vec![Lexeme { token: Token::String("div".into()), offset: Some(0) }],
            children: vec![Node::Rule(middle_rule)],
        };

        assert_eq!(
            outer_rule.optimize(),
            vec![
                SassRule {
                    selectors: vec![
                        Lexeme { token: Token::String("div span img".into()), offset: Some(0) },
                    ],
                    children: vec![
                        Node::Property(
                            Lexeme { token: Token::String("color".into()), offset: Some(12) },
                            Expression::String(
                                Lexeme { token: Token::String("blue".into()), offset: Some(19) }
                            ),
                        ),
                        Node::Rule(
                            SassRule {
                                selectors: vec![
                                    Lexeme {
                                        token: Token::String("strong".into()),
                                        offset: Some(6)
                                    },
                                ],
                                children: vec![Node::Property(
                                    Lexeme { token: Token::String("font-weight".into()), offset: Some(12) },
                                    Expression::String(
                                        Lexeme { token: Token::String("bold".into()), offset: Some(19) }
                                    ),
                                )],
                            }
                        ),
                    ],
                },
            ]
        );
    }
}
<|endoftext|>


<commit_before>use crate::argv::*;
use crate::error::ErrorKind;
use crate::test::helper::get_root_dir;
use std::env;
use std::path::Path;

// TODO: This test only consider the repository name 'rhysd/git-brws.git'

#[test]
fn args_with_no_option() {
    match Parsed::parse_iter(&["git-brws"]).unwrap() {
        Parsed::OpenPage(c) => {
            assert!(
                &[
                    "https://github.com/rhysd/git-brws.git",
                    "ssh://git@github.com:22/rhysd/git-brws.git",
                    "git@github.com:rhysd/git-brws.git",
                    // On GitHub, omitting '.git' is ok for Git URLs
                    "https://github.com/rhysd/git-brws",
                ]
                .contains(&c.repo_url.as_str()),
                "{:?}",
                c.repo_url,
            );
            assert_eq!(c.branch, None);
            assert_eq!(c.cwd, env::current_dir().unwrap().canonicalize().unwrap());
            assert!(c.args.is_empty());
            assert!(!c.stdout);
            assert!(!c.website);
            assert!(!c.blame);
        }
        r => assert!(false, "Failed to parse args with no option: {:?}", r),
    };

    match Parsed::parse_iter(&["git-brws", "foo", "bar"]).unwrap() {
        Parsed::OpenPage(c) => {
            assert_eq!(c.args.len(), 2);
        }
        p => assert!(false, "{:?}", p),
    };
}

#[test]
fn multiple_options() {
    let dir = Path::new(file!()).parent().unwrap();
    match Parsed::parse_iter(&[
        "git-brws",
        "-u",
        "-r",
        "foo/bar",
        "--dir",
        dir.to_str().unwrap(),
        "-b",
        "dev",
        "-w",
        "--blame",
    ])
    .unwrap()
    {
        Parsed::OpenPage(c) => {
            assert_eq!(c.repo_url, "https://github.com/foo/bar.git");
            assert_eq!(c.branch, Some("dev".to_string()));
            assert_eq!(c.cwd, dir.canonicalize().unwrap());
            assert_eq!(c.args.len(), 0);
            assert!(c.stdout);
            assert!(c.website);
            assert!(c.blame);
        }
        p => assert!(false, "{:?}", p),
    };
}

#[test]
fn fix_ssh_repo_url() {
    for (url, expected) in &[
        // GitHub SSH protocols with SCP form.
        // .git file extension and ssh:// and port number can be omitted
        (
            "git@github.com:user/repo.git",
            "ssh://git@github.com:22/user/repo.git",
        ),
        (
            "ssh://git@github.com:user/repo.git",
            "ssh://git@github.com:22/user/repo.git",
        ),
        (
            "ssh://git@github.com:22/user/repo.git",
            "ssh://git@github.com:22/user/repo.git",
        ),
        (
            "ssh://git@github.com:user/repo",
            "ssh://git@github.com:22/user/repo.git",
        ),
        // Azure DevOps URLs
        (
            "ssh://team@vs-ssh.visualstudio.com:v3/team/repo/repo",
            "ssh://team@vs-ssh.visualstudio.com:22/v3/team/repo/repo.git",
        ),
        (
            "ssh://git@ssh.dev.azure.com:v3/team/repo/repo",
            "ssh://git@ssh.dev.azure.com:22/v3/team/repo/repo.git",
        ),
        // Port number is not omitted
        (
            "git@github.somewhere.com:123/user/repo.git",
            "ssh://git@github.somewhere.com:123/user/repo.git",
        ),
    ] {
        match Parsed::parse_iter(&["git-brws", "-r", url]).unwrap() {
            Parsed::OpenPage(c) => {
                assert_eq!(c.repo_url, *expected);
            }
            p => assert!(
                false,
                "Parse must be succeeded but actually results in {:?}",
                p
            ),
        }
    }
}

#[test]
fn repo_formatting() {
    let p = |r| Parsed::parse_iter(&["git-brws", "-r", r]).unwrap();
    match p("bitbucket.org/foo/bar") {
        Parsed::OpenPage(c) => assert_eq!(c.repo_url, "https://bitbucket.org/foo/bar.git"),
        p => assert!(false, "{:?}", p),
    }
    match p("https://gitlab.com/foo/bar") {
        Parsed::OpenPage(c) => assert_eq!(c.repo_url, "https://gitlab.com/foo/bar.git"),
        p => assert!(false, "{:?}", p),
    }
    match p("foo/bar") {
        Parsed::OpenPage(c) => assert_eq!(c.repo_url, "https://github.com/foo/bar.git"),
        p => assert!(false, "{:?}", p),
    }
}

#[test]
fn valid_remote_name() {
    match Parsed::parse_iter(&["git-brws", "-R", "origin"]).unwrap() {
        Parsed::OpenPage(c) => assert!(
            [
                "https://github.com/rhysd/git-brws.git",
                // On GitHub, omitting '.git' is ok for Git URLs
                "https://github.com/rhysd/git-brws",
                "ssh://git@github.com:22/rhysd/git-brws.git"
            ]
            .contains(&c.repo_url.as_str()),
            "Unexpected remote URL for 'origin' remote: {}. For pull request, please ignore this test is failing",
            c.repo_url,
        ),
        p => assert!(false, "{:?}", p),
    }
}

#[test]
fn invalid_remote_name() {
    match Parsed::parse_iter(&["git-brws", "-R", "this-remote-is-never-existing"])
        .unwrap_err()
        .kind()
    {
        ErrorKind::GitObjectNotFound { kind, object, .. } => {
            assert_eq!(*kind, "remote");
            assert_eq!(object, "this-remote-is-never-existing");
        }
        e => assert!(false, "Unexpected error: {}", e),
    }
}

#[test]
fn help_option() {
    match Parsed::parse_iter(&["git-brws", "-h"]).unwrap() {
        Parsed::Help(s) => {
            assert!(s.starts_with("Usage:"));
        }
        p => assert!(false, "{:?}", p),
    }
}

#[test]
fn version_option() {
    match Parsed::parse_iter(&["git-brws", "-v"]).unwrap() {
        Parsed::Version(s) => {
            assert!(!s.is_empty());
        }
        p => assert!(false, "{:?}", p),
    }
}

#[test]
fn unknown_options() {
    assert!(Parsed::parse_iter(&["git-brws", "--unknown"]).is_err());
}

// For checking #9
#[test]
fn specify_repo_outside_repository() {
    let root = get_root_dir();
    match Parsed::parse_iter(&["git-brws", "-d", root.to_str().unwrap(), "-r", "foo/bar"]).unwrap()
    {
        Parsed::OpenPage(c) => {
            assert_eq!(c.cwd, root);
            assert_eq!(&c.repo_url, "https://github.com/foo/bar.git");
        }
        p => assert!(false, "{:?}", p),
    }
}

#[test]
fn search_repo_from_github_by_name() {
    skip_if_no_token_for_search!();

    // Add user:rhysd to ensure to get expected result. But actually repository name is usually
    // passed like `-r react` as use case.
    let parsed = Parsed::parse_iter(&["git-brws", "-r", "user:rhysd vim.wasm"]).unwrap();
    match parsed {
        Parsed::OpenPage(c) => {
            assert_eq!(&c.repo_url, "https://github.com/rhysd/vim.wasm.git");
        }
        p => assert!(false, "{:?}", p),
    }
}

#[test]
fn repo_specified_but_argument_is_not_empty() {
    let err = Parsed::parse_iter(&["git-brws", "-r", "foo", "HEAD"]).unwrap_err();
    match err.kind() {
        ErrorKind::ArgsNotAllowed { ref args, .. } => {
            assert!(format!("{}", err).contains("\"HEAD\""), "{:?}", args);
        }
        e => assert!(false, "Unexpected error: {}", e),
    }
}
<commit_msg>add tests for #24 edge case and SCP-like syntax with port number<commit_after>use crate::argv::*;
use crate::error::ErrorKind;
use crate::test::helper::get_root_dir;
use std::env;
use std::path::Path;

// TODO: This test only consider the repository name 'rhysd/git-brws.git'

#[test]
fn args_with_no_option() {
    match Parsed::parse_iter(&["git-brws"]).unwrap() {
        Parsed::OpenPage(c) => {
            assert!(
                &[
                    "https://github.com/rhysd/git-brws.git",
                    "ssh://git@github.com:22/rhysd/git-brws.git",
                    "git@github.com:rhysd/git-brws.git",
                    // On GitHub, omitting '.git' is ok for Git URLs
                    "https://github.com/rhysd/git-brws",
                ]
                .contains(&c.repo_url.as_str()),
                "{:?}",
                c.repo_url,
            );
            assert_eq!(c.branch, None);
            assert_eq!(c.cwd, env::current_dir().unwrap().canonicalize().unwrap());
            assert!(c.args.is_empty());
            assert!(!c.stdout);
            assert!(!c.website);
            assert!(!c.blame);
        }
        r => assert!(false, "Failed to parse args with no option: {:?}", r),
    };

    match Parsed::parse_iter(&["git-brws", "foo", "bar"]).unwrap() {
        Parsed::OpenPage(c) => {
            assert_eq!(c.args.len(), 2);
        }
        p => assert!(false, "{:?}", p),
    };
}

#[test]
fn multiple_options() {
    let dir = Path::new(file!()).parent().unwrap();
    match Parsed::parse_iter(&[
        "git-brws",
        "-u",
        "-r",
        "foo/bar",
        "--dir",
        dir.to_str().unwrap(),
        "-b",
        "dev",
        "-w",
        "--blame",
    ])
    .unwrap()
    {
        Parsed::OpenPage(c) => {
            assert_eq!(c.repo_url, "https://github.com/foo/bar.git");
            assert_eq!(c.branch, Some("dev".to_string()));
            assert_eq!(c.cwd, dir.canonicalize().unwrap());
            assert_eq!(c.args.len(), 0);
            assert!(c.stdout);
            assert!(c.website);
            assert!(c.blame);
        }
        p => assert!(false, "{:?}", p),
    };
}

#[test]
fn fix_ssh_repo_url() {
    for (url, expected) in &[
        // GitHub SSH protocols with SCP form.
        // .git file extension and ssh:// and port number can be omitted
        (
            "git@github.com:user/repo.git",
            "ssh://git@github.com:22/user/repo.git",
        ),
        (
            "ssh://git@github.com/user/repo.git",
            "ssh://git@github.com/user/repo.git",
        ),
        (
            "ssh://git@github.com:22/user/repo.git",
            "ssh://git@github.com:22/user/repo.git",
        ),
        // Azure DevOps URLs
        (
            "team@vs-ssh.visualstudio.com:v3/team/repo/repo",
            "ssh://team@vs-ssh.visualstudio.com:22/v3/team/repo/repo.git",
        ),
        (
            "git@ssh.dev.azure.com:v3/team/repo/repo",
            "ssh://git@ssh.dev.azure.com:22/v3/team/repo/repo.git",
        ),
        // Port number is not omitted
        (
            "git@github.somewhere.com:123:user/repo.git",
            "ssh://git@github.somewhere.com:123/user/repo.git",
        ),
        // Issue #24. When a user name is integer
        (
            "git@github.com:1234/repo.git",
            "ssh://git@github.com:22/1234/repo.git",
        ),
        (
            "git@github.com:0x7FFFFFFFFFFFFFFF/repo.git",
            "ssh://git@github.com:22/0x7FFFFFFFFFFFFFFF/repo.git",
        ),
        (
            "git@github.somewhere.com:123:456/repo.git",
            "ssh://git@github.somewhere.com:123/456/repo.git",
        ),
    ] {
        match Parsed::parse_iter(&["git-brws", "-r", url]).unwrap() {
            Parsed::OpenPage(c) => {
                assert_eq!(c.repo_url, *expected);
            }
            p => assert!(
                false,
                "Parse must be succeeded but actually results in {:?}",
                p
            ),
        }
    }
}

#[test]
fn repo_formatting() {
    let p = |r| Parsed::parse_iter(&["git-brws", "-r", r]).unwrap();
    match p("bitbucket.org/foo/bar") {
        Parsed::OpenPage(c) => assert_eq!(c.repo_url, "https://bitbucket.org/foo/bar.git"),
        p => assert!(false, "{:?}", p),
    }
    match p("https://gitlab.com/foo/bar") {
        Parsed::OpenPage(c) => assert_eq!(c.repo_url, "https://gitlab.com/foo/bar.git"),
        p => assert!(false, "{:?}", p),
    }
    match p("foo/bar") {
        Parsed::OpenPage(c) => assert_eq!(c.repo_url, "https://github.com/foo/bar.git"),
        p => assert!(false, "{:?}", p),
    }
}

#[test]
fn valid_remote_name() {
    match Parsed::parse_iter(&["git-brws", "-R", "origin"]).unwrap() {
        Parsed::OpenPage(c) => assert!(
            [
                "https://github.com/rhysd/git-brws.git",
                // On GitHub, omitting '.git' is ok for Git URLs
                "https://github.com/rhysd/git-brws",
                "ssh://git@github.com:22/rhysd/git-brws.git"
            ]
            .contains(&c.repo_url.as_str()),
            "Unexpected remote URL for 'origin' remote: {}. For pull request, please ignore this test is failing",
            c.repo_url,
        ),
        p => assert!(false, "{:?}", p),
    }
}

#[test]
fn invalid_remote_name() {
    match Parsed::parse_iter(&["git-brws", "-R", "this-remote-is-never-existing"])
        .unwrap_err()
        .kind()
    {
        ErrorKind::GitObjectNotFound { kind, object, .. } => {
            assert_eq!(*kind, "remote");
            assert_eq!(object, "this-remote-is-never-existing");
        }
        e => assert!(false, "Unexpected error: {}", e),
    }
}

#[test]
fn help_option() {
    match Parsed::parse_iter(&["git-brws", "-h"]).unwrap() {
        Parsed::Help(s) => {
            assert!(s.starts_with("Usage:"));
        }
        p => assert!(false, "{:?}", p),
    }
}

#[test]
fn version_option() {
    match Parsed::parse_iter(&["git-brws", "-v"]).unwrap() {
        Parsed::Version(s) => {
            assert!(!s.is_empty());
        }
        p => assert!(false, "{:?}", p),
    }
}

#[test]
fn unknown_options() {
    assert!(Parsed::parse_iter(&["git-brws", "--unknown"]).is_err());
}

// For checking #9
#[test]
fn specify_repo_outside_repository() {
    let root = get_root_dir();
    match Parsed::parse_iter(&["git-brws", "-d", root.to_str().unwrap(), "-r", "foo/bar"]).unwrap()
    {
        Parsed::OpenPage(c) => {
            assert_eq!(c.cwd, root);
            assert_eq!(&c.repo_url, "https://github.com/foo/bar.git");
        }
        p => assert!(false, "{:?}", p),
    }
}

#[test]
fn search_repo_from_github_by_name() {
    skip_if_no_token_for_search!();

    // Add user:rhysd to ensure to get expected result. But actually repository name is usually
    // passed like `-r react` as use case.
    let parsed = Parsed::parse_iter(&["git-brws", "-r", "user:rhysd vim.wasm"]).unwrap();
    match parsed {
        Parsed::OpenPage(c) => {
            assert_eq!(&c.repo_url, "https://github.com/rhysd/vim.wasm.git");
        }
        p => assert!(false, "{:?}", p),
    }
}

#[test]
fn repo_specified_but_argument_is_not_empty() {
    let err = Parsed::parse_iter(&["git-brws", "-r", "foo", "HEAD"]).unwrap_err();
    match err.kind() {
        ErrorKind::ArgsNotAllowed { ref args, .. } => {
            assert!(format!("{}", err).contains("\"HEAD\""), "{:?}", args);
        }
        e => assert!(false, "Unexpected error: {}", e),
    }
}
<|endoftext|>


<commit_before>use token::{Lexeme, Token};
use error::{Result, SassError, ErrorKind};

use std::str::CharIndices;
use std::iter::Peekable;

pub struct Tokenizer<'a> {
    text: &'a str,
    chars: Peekable<CharIndices<'a>>,
}

impl<'a> Iterator for Tokenizer<'a> {
    type Item = Result<Lexeme>;

    fn next(&mut self) -> Option<Result<Lexeme>> {
        match self.parse() {
            Ok(Some(t)) => Some(Ok(t)),
            Ok(None) => None,
            Err(e) => Some(Err(e)),
        }
    }
}

impl<'a> Tokenizer<'a> {
    pub fn new(text: &str) -> Tokenizer {
        Tokenizer {
            text: text,
            chars: text.char_indices().peekable(),
        }
    }

    pub fn parse(&mut self) -> Result<Option<Lexeme>> {
        while let Some((char_offset, curr_char)) = self.chars.next() {
            // Skip leading whitespace
            if curr_char.is_whitespace() {
                continue;
            } else {
                let single_char_token = Token::from_char(curr_char);
                if single_char_token.is_some() {
                    if curr_char == '-' {
                        if let Some(&(_, peek_char)) = self.chars.peek() {
                            if peek_char.is_whitespace() {
                                return Ok(Some(Lexeme {
                                    token: Token::Minus,
                                    offset: Some(char_offset)
                                }))
                            } else if peek_char.is_numeric() {
                                return self.number(curr_char, char_offset)
                            } else {
                                return self.ident(curr_char, char_offset)
                            }
                        }
                    } else {
                        // We already tested that single_char_token was Some.
                        return Ok(Some(Lexeme {
                            token: single_char_token.unwrap(),
                            offset: Some(char_offset),
                        }))
                    }
                } else {
                    if curr_char.is_numeric() {
                        return self.number(curr_char, char_offset)
                    } else {
                        return self.ident(curr_char, char_offset)
                    }
                }
            }
        }
        return Ok(None)
    }

    fn peek_char(&mut self) -> Option<char> {
        match self.chars.peek() {
            Some(&(_, peek_char)) => Some(peek_char),
            None => None,
        }
    }

    fn ident(&mut self, curr_char: char, start: usize) -> Result<Option<Lexeme>> {
        let mut value = String::new();
        value.push(curr_char);

        while let Some(peek_char) = self.peek_char() {
            // Stop when we reach a non-ident char (hyphens are special)
            if peek_char.is_whitespace() || (
                is_single_char_token(peek_char) && peek_char != '-'
            ) {
                break;
            } else {
                value.push(peek_char);
                self.chars.next();
            }
        }
        Ok(Some(Lexeme { token: Token::Ident(value), offset: Some(start) }))
    }

    fn number(&mut self, curr_char: char, start: usize) -> Result<Option<Lexeme>> {
        let mut value = String::new();
        value.push(curr_char);

        while let Some(peek_char) = self.peek_char() {
            // Stop when we reach a non-numeric char
            if !peek_char.is_numeric() && peek_char != '.' {
                break;
            } else {
                value.push(peek_char);
                self.chars.next();
            }
        }

        let value = match value.parse() {
            Ok(v) => v,
            Err(_) => return Err(SassError {
                offset: start,
                kind: ErrorKind::TokenizerError,
                message: format!(
                    "Tried to parse `{}` into a f32 but failed.",
                    value,
                ),
            })
        };
        Ok(Some(Lexeme { token: Token::Number(value), offset: Some(start) }))
    }
}

fn is_single_char_token(ch: char) -> bool {
    Token::from_char(ch).is_some()
}

#[cfg(test)]
mod tests {
    use super::*;
    use token::{Lexeme, Token};
    use error::Result;

    fn expected_lexeme(expected_token: Token, expected_offset: usize) -> Option<Result<Lexeme>> {
        Some(Ok(Lexeme {
            token: expected_token,
            offset: Some(expected_offset),
        }))
    }

    fn expected_ident(expected_value: &str, expected_offset: usize) -> Option<Result<Lexeme>> {
        expected_lexeme(Token::Ident(expected_value.into()), expected_offset)
    }

    #[test]
    fn it_returns_none_for_empty_string() {
        let mut tokenizer = Tokenizer::new("");
        assert_eq!(tokenizer.next(), None);
    }

    #[test]
    fn it_returns_none_for_only_whitespace() {
        let mut tokenizer = Tokenizer::new("     ");
        assert_eq!(tokenizer.next(), None);
    }

    #[test]
    fn it_returns_words() {
        // Without regard to Sass word validity
        let mut tokenizer = Tokenizer::new(" \n  div   aoeu  ");
        assert_eq!(tokenizer.next(), expected_ident("div", 4));
        assert_eq!(tokenizer.next(), expected_ident("aoeu", 10));
        assert_eq!(tokenizer.next(), None);
    }

    #[test]
    fn it_separates_curly_braces() {
        // Without regard to matching
        let mut tokenizer = Tokenizer::new("{}}a{ blah}");
        assert_eq!(tokenizer.next(), expected_lexeme(Token::LeftCurlyBrace, 0));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::RightCurlyBrace, 1));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::RightCurlyBrace, 2));
        assert_eq!(tokenizer.next(), expected_ident("a", 3));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::LeftCurlyBrace, 4));
        assert_eq!(tokenizer.next(), expected_ident("blah", 6));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::RightCurlyBrace, 10));
        assert_eq!(tokenizer.next(), None);
    }

    #[test]
    fn it_separates_colon() {
        let mut tokenizer = Tokenizer::new(":invalid: property::");
        // This might be wrong. I have a note somewhere else that colons can start idents.
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Colon, 0));
        assert_eq!(tokenizer.next(), expected_ident("invalid", 1));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Colon, 8));
        assert_eq!(tokenizer.next(), expected_ident("property", 10));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Colon, 18));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Colon, 19));
        assert_eq!(tokenizer.next(), None);
    }

    #[test]
    fn it_separates_semicolon() {
        let mut tokenizer = Tokenizer::new(";;\na;\nb\n;");
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Semicolon, 0));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Semicolon, 1));
        assert_eq!(tokenizer.next(), expected_ident("a", 3));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Semicolon, 4));
        assert_eq!(tokenizer.next(), expected_ident("b", 6));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Semicolon, 8));
        assert_eq!(tokenizer.next(), None);
    }

    #[test]
    fn it_separates_numbers() {
        let mut tokenizer = Tokenizer::new("border: 0px 1.5 11em;");
        assert_eq!(tokenizer.next(), expected_ident("border", 0));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Colon, 6));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Number(0.0), 8));
        assert_eq!(tokenizer.next(), expected_ident("px", 9));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Number(1.5), 12));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Number(11.0), 16));
        assert_eq!(tokenizer.next(), expected_ident("em", 18));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Semicolon, 20));
        assert_eq!(tokenizer.next(), None);
    }

    #[test]
    fn it_separates_hyphen() {
        let mut tokenizer = Tokenizer::new("font-weight -webkit -3 - 4-5 a-1");
        assert_eq!(tokenizer.next(), expected_ident("font-weight", 0));
        assert_eq!(tokenizer.next(), expected_ident("-webkit", 12));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Number(-3.0), 20));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Minus, 23));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Number(4.0), 25));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Number(-5.0), 26));
        assert_eq!(tokenizer.next(), expected_ident("a-1", 29));
        assert_eq!(tokenizer.next(), None);
    }

    #[test]
    fn it_separates_parens() {
        let mut tokenizer = Tokenizer::new("() rgb)()(");
        assert_eq!(tokenizer.next(), expected_lexeme(Token::LeftParen, 0));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::RightParen, 1));
        assert_eq!(tokenizer.next(), expected_ident("rgb", 3));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::RightParen, 6));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::LeftParen, 7));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::RightParen, 8));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::LeftParen, 9));
        assert_eq!(tokenizer.next(), None);
    }

    #[test]
    fn it_separates_slash() {
        let mut tokenizer = Tokenizer::new("/ / 3/4 / 8");
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Slash, 0));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Slash, 2));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Number(3.0), 4));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Slash, 5));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Number(4.0), 6));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Slash, 8));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Number(8.0), 10));
        assert_eq!(tokenizer.next(), None);
    }
}
<commit_msg>Fix an edge case with hyphen at the end of the input<commit_after>use token::{Lexeme, Token};
use error::{Result, SassError, ErrorKind};

use std::str::CharIndices;
use std::iter::Peekable;

pub struct Tokenizer<'a> {
    text: &'a str,
    chars: Peekable<CharIndices<'a>>,
}

impl<'a> Iterator for Tokenizer<'a> {
    type Item = Result<Lexeme>;

    fn next(&mut self) -> Option<Result<Lexeme>> {
        match self.parse() {
            Ok(Some(t)) => Some(Ok(t)),
            Ok(None) => None,
            Err(e) => Some(Err(e)),
        }
    }
}

impl<'a> Tokenizer<'a> {
    pub fn new(text: &str) -> Tokenizer {
        Tokenizer {
            text: text,
            chars: text.char_indices().peekable(),
        }
    }

    pub fn parse(&mut self) -> Result<Option<Lexeme>> {
        while let Some((char_offset, curr_char)) = self.chars.next() {
            // Skip leading whitespace
            if curr_char.is_whitespace() {
                continue;
            } else {
                let single_char_token = Token::from_char(curr_char);
                if single_char_token.is_some() {
                    let peek_char = self.peek_char();
                    if curr_char == '-' &&
                       peek_char.is_some() &&
                       !peek_char.unwrap().is_whitespace() {

                        if peek_char.unwrap().is_numeric() {
                            return self.number(curr_char, char_offset)
                        } else {
                            return self.ident(curr_char, char_offset)
                        }

                    } else {
                        // We already tested that single_char_token was Some.
                        return Ok(Some(Lexeme {
                            token: single_char_token.unwrap(),
                            offset: Some(char_offset),
                        }))
                    }
                } else {
                    if curr_char.is_numeric() {
                        return self.number(curr_char, char_offset)
                    } else {
                        return self.ident(curr_char, char_offset)
                    }
                }
            }
        }
        return Ok(None)
    }

    fn peek_char(&mut self) -> Option<char> {
        match self.chars.peek() {
            Some(&(_, peek_char)) => Some(peek_char),
            None => None,
        }
    }

    fn ident(&mut self, curr_char: char, start: usize) -> Result<Option<Lexeme>> {
        let mut value = String::new();
        value.push(curr_char);

        while let Some(peek_char) = self.peek_char() {
            // Stop when we reach a non-ident char (hyphens are special)
            if peek_char.is_whitespace() || (
                is_single_char_token(peek_char) && peek_char != '-'
            ) {
                break;
            } else {
                value.push(peek_char);
                self.chars.next();
            }
        }
        Ok(Some(Lexeme { token: Token::Ident(value), offset: Some(start) }))
    }

    fn number(&mut self, curr_char: char, start: usize) -> Result<Option<Lexeme>> {
        let mut value = String::new();
        value.push(curr_char);

        while let Some(peek_char) = self.peek_char() {
            // Stop when we reach a non-numeric char
            if !peek_char.is_numeric() && peek_char != '.' {
                break;
            } else {
                value.push(peek_char);
                self.chars.next();
            }
        }

        let value = match value.parse() {
            Ok(v) => v,
            Err(_) => return Err(SassError {
                offset: start,
                kind: ErrorKind::TokenizerError,
                message: format!(
                    "Tried to parse `{}` into a f32 but failed.",
                    value,
                ),
            })
        };
        Ok(Some(Lexeme { token: Token::Number(value), offset: Some(start) }))
    }
}

fn is_single_char_token(ch: char) -> bool {
    Token::from_char(ch).is_some()
}

#[cfg(test)]
mod tests {
    use super::*;
    use token::{Lexeme, Token};
    use error::Result;

    fn expected_lexeme(expected_token: Token, expected_offset: usize) -> Option<Result<Lexeme>> {
        Some(Ok(Lexeme {
            token: expected_token,
            offset: Some(expected_offset),
        }))
    }

    fn expected_ident(expected_value: &str, expected_offset: usize) -> Option<Result<Lexeme>> {
        expected_lexeme(Token::Ident(expected_value.into()), expected_offset)
    }

    #[test]
    fn it_returns_none_for_empty_string() {
        let mut tokenizer = Tokenizer::new("");
        assert_eq!(tokenizer.next(), None);
    }

    #[test]
    fn it_returns_none_for_only_whitespace() {
        let mut tokenizer = Tokenizer::new("     ");
        assert_eq!(tokenizer.next(), None);
    }

    #[test]
    fn it_returns_words() {
        // Without regard to Sass word validity
        let mut tokenizer = Tokenizer::new(" \n  div   aoeu  ");
        assert_eq!(tokenizer.next(), expected_ident("div", 4));
        assert_eq!(tokenizer.next(), expected_ident("aoeu", 10));
        assert_eq!(tokenizer.next(), None);
    }

    #[test]
    fn it_separates_curly_braces() {
        // Without regard to matching
        let mut tokenizer = Tokenizer::new("{}}a{ blah}");
        assert_eq!(tokenizer.next(), expected_lexeme(Token::LeftCurlyBrace, 0));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::RightCurlyBrace, 1));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::RightCurlyBrace, 2));
        assert_eq!(tokenizer.next(), expected_ident("a", 3));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::LeftCurlyBrace, 4));
        assert_eq!(tokenizer.next(), expected_ident("blah", 6));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::RightCurlyBrace, 10));
        assert_eq!(tokenizer.next(), None);
    }

    #[test]
    fn it_separates_colon() {
        let mut tokenizer = Tokenizer::new(":invalid: property::");
        // This might be wrong. I have a note somewhere else that colons can start idents.
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Colon, 0));
        assert_eq!(tokenizer.next(), expected_ident("invalid", 1));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Colon, 8));
        assert_eq!(tokenizer.next(), expected_ident("property", 10));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Colon, 18));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Colon, 19));
        assert_eq!(tokenizer.next(), None);
    }

    #[test]
    fn it_separates_semicolon() {
        let mut tokenizer = Tokenizer::new(";;\na;\nb\n;");
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Semicolon, 0));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Semicolon, 1));
        assert_eq!(tokenizer.next(), expected_ident("a", 3));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Semicolon, 4));
        assert_eq!(tokenizer.next(), expected_ident("b", 6));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Semicolon, 8));
        assert_eq!(tokenizer.next(), None);
    }

    #[test]
    fn it_separates_numbers() {
        let mut tokenizer = Tokenizer::new("border: 0px 1.5 11em;");
        assert_eq!(tokenizer.next(), expected_ident("border", 0));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Colon, 6));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Number(0.0), 8));
        assert_eq!(tokenizer.next(), expected_ident("px", 9));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Number(1.5), 12));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Number(11.0), 16));
        assert_eq!(tokenizer.next(), expected_ident("em", 18));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Semicolon, 20));
        assert_eq!(tokenizer.next(), None);
    }

    #[test]
    fn it_separates_hyphen() {
        let mut tokenizer = Tokenizer::new("font-weight -webkit -3 - 4-5 a-1 -");
        assert_eq!(tokenizer.next(), expected_ident("font-weight", 0));
        assert_eq!(tokenizer.next(), expected_ident("-webkit", 12));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Number(-3.0), 20));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Minus, 23));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Number(4.0), 25));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Number(-5.0), 26));
        assert_eq!(tokenizer.next(), expected_ident("a-1", 29));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Minus, 33));
        assert_eq!(tokenizer.next(), None);
    }

    #[test]
    fn it_separates_parens() {
        let mut tokenizer = Tokenizer::new("() rgb)()(");
        assert_eq!(tokenizer.next(), expected_lexeme(Token::LeftParen, 0));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::RightParen, 1));
        assert_eq!(tokenizer.next(), expected_ident("rgb", 3));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::RightParen, 6));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::LeftParen, 7));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::RightParen, 8));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::LeftParen, 9));
        assert_eq!(tokenizer.next(), None);
    }

    #[test]
    fn it_separates_slash() {
        let mut tokenizer = Tokenizer::new("/ / 3/4 / 8");
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Slash, 0));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Slash, 2));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Number(3.0), 4));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Slash, 5));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Number(4.0), 6));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Slash, 8));
        assert_eq!(tokenizer.next(), expected_lexeme(Token::Number(8.0), 10));
        assert_eq!(tokenizer.next(), None);
    }
}
<|endoftext|>


<commit_before>// Copyright (c) 2017-2018, The rav1e contributors. All rights reserved
//
// This source code is subject to the terms of the BSD 2 Clause License and
// the Alliance for Open Media Patent License 1.0. If the BSD 2 Clause License
// was not distributed with this source code in the LICENSE file, you can
// obtain it at www.aomedia.org/license/software. If the Alliance for Open
// Media Patent License 1.0 was not distributed with this source code in the
// PATENTS file, you can obtain it at www.aomedia.org/license/patent.

extern crate libc;

use partition::TxSize;
use partition::TxType;

// In libaom, functions that have more than one specialization use function
// pointers, so we need to declare them as static fields and call them
// indirectly. Otherwise, we call SSE or C variants directly. To fully
// understand what's going on here you should first understand the Perl code
// in libaom that generates these function bindings.

extern {
  fn av1_fwd_txfm2d_4x4_c(
    input: *const i16, output: *mut i32, stride: libc::c_int,
    tx_type: libc::c_int, bd: libc::c_int
  );
  fn av1_fwd_txfm2d_8x8_c(
    input: *const i16, output: *mut i32, stride: libc::c_int,
    tx_type: libc::c_int, bd: libc::c_int
  );
  fn av1_fwd_txfm2d_16x16_c(
    input: *const i16, output: *mut i32, stride: libc::c_int,
    tx_type: libc::c_int, bd: libc::c_int
  );
  fn av1_fwd_txfm2d_32x32_c(
    input: *const i16, output: *mut i32, stride: libc::c_int,
    tx_type: libc::c_int, bd: libc::c_int
  );
}

extern "C" {
  static av1_inv_txfm2d_add_4x4: extern fn(
    input: *const i32,
    output: *mut u16,
    stride: libc::c_int,
    tx_type: libc::c_int,
    bd: libc::c_int
  );
  fn av1_inv_txfm2d_add_4x4_c(
    input: *const i32,
    output: *mut u16,
    stride: libc::c_int,
    tx_type: libc::c_int,
    bd: libc::c_int
  );
  static av1_inv_txfm2d_add_8x8: extern fn(
    input: *const i32,
    output: *mut u16,
    stride: libc::c_int,
    tx_type: libc::c_int,
    bd: libc::c_int
  ) -> ();
  fn av1_inv_txfm2d_add_8x8_c(
    input: *const i32,
    output: *mut u16,
    stride: libc::c_int,
    tx_type: libc::c_int,
    bd: libc::c_int
  ) -> ();
  static av1_inv_txfm2d_add_16x16: extern fn(
    input: *const i32,
    output: *mut u16,
    stride: libc::c_int,
    tx_type: libc::c_int,
    bd: libc::c_int
  );
  fn av1_inv_txfm2d_add_16x16_c(
    input: *const i32, output: *mut u16, stride: libc::c_int,
    tx_type: libc::c_int, bd: libc::c_int
  );
  static av1_inv_txfm2d_add_32x32: extern fn(
    input: *const i32,
    output: *mut u16,
    stride: libc::c_int,
    tx_type: libc::c_int,
    bd: libc::c_int
  );
  fn av1_inv_txfm2d_add_32x32_c(
    input: *const i32,
    output: *mut u16,
    stride: libc::c_int,
    tx_type: libc::c_int,
    bd: libc::c_int
  );
}

pub fn forward_transform(
  input: &[i16], output: &mut [i32], stride: usize, tx_size: TxSize,
  tx_type: TxType, bit_depth: usize
) {
  match tx_size {
    TxSize::TX_4X4 => fht4x4(input, output, stride, tx_type, bit_depth),
    TxSize::TX_8X8 => fht8x8(input, output, stride, tx_type, bit_depth),
    TxSize::TX_16X16 => fht16x16(input, output, stride, tx_type, bit_depth),
    TxSize::TX_32X32 => fht32x32(input, output, stride, tx_type, bit_depth),
    _ => panic!("unimplemented tx size")
  }
}

pub fn inverse_transform_add(
  input: &[i32], output: &mut [u16], stride: usize, tx_size: TxSize,
  tx_type: TxType, bit_depth: usize
) {
  match tx_size {
    TxSize::TX_4X4 => iht4x4_add(input, output, stride, tx_type, bit_depth),
    TxSize::TX_8X8 => iht8x8_add(input, output, stride, tx_type, bit_depth),
    TxSize::TX_16X16 => iht16x16_add(input, output, stride, tx_type, bit_depth),
    TxSize::TX_32X32 => iht32x32_add(input, output, stride, tx_type, bit_depth),
    _ => panic!("unimplemented tx size")
  }
}

fn fht4x4(input: &[i16], output: &mut [i32], stride: usize, tx_type: TxType, bit_depth: usize) {
  unsafe {
    av1_fwd_txfm2d_4x4_c(
      input.as_ptr(),
      output.as_mut_ptr(),
      stride as libc::c_int,
      tx_type as libc::c_int,
      bit_depth as libc::c_int
    );
  }
}

fn iht4x4_add(
  input: &[i32], output: &mut [u16], stride: usize, tx_type: TxType, bit_depth: usize
) {
  // SIMD code may assert for transform types beyond TxType::IDTX.
  if tx_type < TxType::IDTX {
    unsafe {
      av1_inv_txfm2d_add_4x4(
        input.as_ptr(),
        output.as_mut_ptr(),
        stride as libc::c_int,
        tx_type as libc::c_int,
        bit_depth as libc::c_int
      );
    }
  } else {
    unsafe {
      av1_inv_txfm2d_add_4x4_c(
        input.as_ptr(),
        output.as_mut_ptr(),
        stride as libc::c_int,
        tx_type as libc::c_int,
        bit_depth as libc::c_int
      );
    }
  }
}

fn fht8x8(input: &[i16], output: &mut [i32], stride: usize, tx_type: TxType, bit_depth: usize) {
  unsafe {
    av1_fwd_txfm2d_8x8_c(
      input.as_ptr(),
      output.as_mut_ptr(),
      stride as libc::c_int,
      tx_type as libc::c_int,
      bit_depth as libc::c_int
    );
  }
}

fn iht8x8_add(
  input: &[i32], output: &mut [u16], stride: usize, tx_type: TxType, bit_depth: usize
) {
  // SIMD code may assert for transform types beyond TxType::IDTX.
  if tx_type < TxType::IDTX {
    unsafe {
      av1_inv_txfm2d_add_8x8(
        input.as_ptr(),
        output.as_mut_ptr(),
        stride as libc::c_int,
        tx_type as libc::c_int,
        bit_depth as libc::c_int
      );
    }
  } else {
    unsafe {
      av1_inv_txfm2d_add_8x8_c(
        input.as_ptr(),
        output.as_mut_ptr(),
        stride as libc::c_int,
        tx_type as libc::c_int,
        bit_depth as libc::c_int
      );
    }
  }
}

fn fht16x16(
  input: &[i16], output: &mut [i32], stride: usize, tx_type: TxType, bit_depth: usize
) {
  unsafe {
    av1_fwd_txfm2d_16x16_c(
      input.as_ptr(),
      output.as_mut_ptr(),
      stride as libc::c_int,
      tx_type as libc::c_int,
      bit_depth as libc::c_int
    );
  }
}

fn iht16x16_add(
  input: &[i32], output: &mut [u16], stride: usize, tx_type: TxType, bit_depth: usize
) {
  unsafe {
    if tx_type < TxType::IDTX {
      // SSE C code asserts for transform types beyond TxType::IDTX.
      av1_inv_txfm2d_add_16x16(
        input.as_ptr(),
        output.as_mut_ptr(),
        stride as libc::c_int,
        tx_type as libc::c_int,
        bit_depth as libc::c_int
      );
    } else {
      av1_inv_txfm2d_add_16x16_c(
        input.as_ptr(),
        output.as_mut_ptr(),
        stride as libc::c_int,
        tx_type as libc::c_int,
        bit_depth as libc::c_int
      );
    }
  }
}

fn fht32x32(
  input: &[i16], output: &mut [i32], stride: usize, tx_type: TxType, bit_depth: usize
) {
  unsafe {
    av1_fwd_txfm2d_32x32_c(
      input.as_ptr(),
      output.as_mut_ptr(),
      stride as libc::c_int,
      tx_type as libc::c_int,
      bit_depth as libc::c_int
    );
  }
}

fn iht32x32_add(
  input: &[i32], output: &mut [u16], stride: usize, tx_type: TxType, bit_depth: usize
) {
  unsafe {
    if tx_type < TxType::IDTX {
      // SIMDI code may assert for transform types beyond TxType::IDTX.
      av1_inv_txfm2d_add_32x32(
        input.as_ptr(),
        output.as_mut_ptr(),
        stride as libc::c_int,
        tx_type as libc::c_int,
        bit_depth as libc::c_int
      );
    } else {
      av1_inv_txfm2d_add_32x32_c(
        input.as_ptr(),
        output.as_mut_ptr(),
        stride as libc::c_int,
        tx_type as libc::c_int,
        bit_depth as libc::c_int
      );
    }
  }
}
<commit_msg>Implement idct 4x4 in rust. (#464)<commit_after>// Copyright (c) 2017-2018, The rav1e contributors. All rights reserved
//
// This source code is subject to the terms of the BSD 2 Clause License and
// the Alliance for Open Media Patent License 1.0. If the BSD 2 Clause License
// was not distributed with this source code in the LICENSE file, you can
// obtain it at www.aomedia.org/license/software. If the Alliance for Open
// Media Patent License 1.0 was not distributed with this source code in the
// PATENTS file, you can obtain it at www.aomedia.org/license/patent.

extern crate libc;

use std::cmp;

use partition::TxSize;
use partition::TxType;

use util::*;

static COSPI_INV: [i32; 64] = [
  4096, 4095, 4091, 4085, 4076, 4065, 4052, 4036, 4017, 3996, 3973, 3948,
  3920, 3889, 3857, 3822, 3784, 3745, 3703, 3659, 3612, 3564, 3513, 3461,
  3406, 3349, 3290, 3229, 3166, 3102, 3035, 2967, 2896, 2824, 2751, 2675,
  2598, 2520, 2440, 2359, 2276, 2191, 2106, 2019, 1931, 1842, 1751, 1660,
  1567, 1474, 1380, 1285, 1189, 1092, 995, 897, 799, 700, 601, 501, 401, 301,
  201, 101,
];

// performs half a butterfly
#[inline]
fn half_btf(w0: i32, in0: i32, w1: i32, in1: i32, bit: usize) -> i32 {
  let result = (w0 * in0) + (w1 * in1);
  round_shift(result, bit)
}

#[inline]
fn round_shift(value: i32, bit: usize) -> i32 {
  (value + (1 << (bit - 1))) >> bit
}

// clamps value to a signed integer type of bit bits
#[inline]
fn clamp_value(value: i32, bit: usize) -> i32 {
  let max_value: i32 = ((1i64 << (bit - 1)) - 1) as i32;
  let min_value: i32 = (-(1i64 << (bit - 1))) as i32;
  clamp(value, min_value, max_value)
}

fn av1_idct4(input: [i32; 4], output: &mut [i32], range: usize) {
  let cos_bit = 12;
  // stage 0

  // stage 1
  let stg1 = [input[0], input[2], input[1], input[3]];

  // stage 2
  let stg2 = [
    half_btf(COSPI_INV[32], stg1[0], COSPI_INV[32], stg1[1], cos_bit),
    half_btf(COSPI_INV[32], stg1[0], -COSPI_INV[32], stg1[1], cos_bit),
    half_btf(COSPI_INV[48], stg1[2], -COSPI_INV[16], stg1[3], cos_bit),
    half_btf(COSPI_INV[16], stg1[2], COSPI_INV[48], stg1[3], cos_bit)
  ];

  // stage 3
  output[0] = clamp_value(stg2[0] + stg2[3], range);
  output[1] = clamp_value(stg2[1] + stg2[2], range);
  output[2] = clamp_value(stg2[1] - stg2[2], range);
  output[3] = clamp_value(stg2[0] - stg2[3], range);
}

fn inv_txfm2d_add_4x4_rs(
  input: &[i32], output: &mut [u16], stride: usize, bd: usize
) {
  let mut buffer = [0i32; 4 * 4];
  // perform inv txfm on every row
  let range = bd + 8;
  for (input_slice, buffer_slice) in input.chunks(4).zip(buffer.chunks_mut(4))
  {
    let mut temp_in: [i32; 4] = [0; 4];
    for (raw, clamped) in input_slice.iter().zip(temp_in.iter_mut()) {
      *clamped = clamp_value(*raw, range);
    }
    av1_idct4(temp_in, buffer_slice, range);
  }

  // perform inv txfm on every col
  let range = cmp::max(bd + 6, 16);
  for c in 0..4 {
    let mut temp_in: [i32; 4] = [0; 4];
    let mut temp_out: [i32; 4] = [0; 4];
    for (raw, clamped) in buffer[c..].iter().step_by(4).zip(temp_in.iter_mut())
    {
      *clamped = clamp_value(*raw, range);
    }
    av1_idct4(temp_in, &mut temp_out, range);
    for (temp, out) in
      temp_out.iter().zip(output[c..].iter_mut().step_by(stride).take(4)) {
      *out =
        clamp(*out as i32 + round_shift(*temp, 4), 0, (1 << bd) - 1) as u16;
    }
  }
}

// In libaom, functions that have more than one specialization use function
// pointers, so we need to declare them as static fields and call them
// indirectly. Otherwise, we call SSE or C variants directly. To fully
// understand what's going on here you should first understand the Perl code
// in libaom that generates these function bindings.

extern {
  fn av1_fwd_txfm2d_4x4_c(
    input: *const i16, output: *mut i32, stride: libc::c_int,
    tx_type: libc::c_int, bd: libc::c_int
  );
  fn av1_fwd_txfm2d_8x8_c(
    input: *const i16, output: *mut i32, stride: libc::c_int,
    tx_type: libc::c_int, bd: libc::c_int
  );
  fn av1_fwd_txfm2d_16x16_c(
    input: *const i16, output: *mut i32, stride: libc::c_int,
    tx_type: libc::c_int, bd: libc::c_int
  );
  fn av1_fwd_txfm2d_32x32_c(
    input: *const i16, output: *mut i32, stride: libc::c_int,
    tx_type: libc::c_int, bd: libc::c_int
  );
}

extern "C" {
  static av1_inv_txfm2d_add_4x4: extern fn(
    input: *const i32,
    output: *mut u16,
    stride: libc::c_int,
    tx_type: libc::c_int,
    bd: libc::c_int
  );
  fn av1_inv_txfm2d_add_4x4_c(
    input: *const i32,
    output: *mut u16,
    stride: libc::c_int,
    tx_type: libc::c_int,
    bd: libc::c_int
  );
  static av1_inv_txfm2d_add_8x8: extern fn(
    input: *const i32,
    output: *mut u16,
    stride: libc::c_int,
    tx_type: libc::c_int,
    bd: libc::c_int
  ) -> ();
  fn av1_inv_txfm2d_add_8x8_c(
    input: *const i32,
    output: *mut u16,
    stride: libc::c_int,
    tx_type: libc::c_int,
    bd: libc::c_int
  ) -> ();
  static av1_inv_txfm2d_add_16x16: extern fn(
    input: *const i32,
    output: *mut u16,
    stride: libc::c_int,
    tx_type: libc::c_int,
    bd: libc::c_int
  );
  fn av1_inv_txfm2d_add_16x16_c(
    input: *const i32, output: *mut u16, stride: libc::c_int,
    tx_type: libc::c_int, bd: libc::c_int
  );
  static av1_inv_txfm2d_add_32x32: extern fn(
    input: *const i32,
    output: *mut u16,
    stride: libc::c_int,
    tx_type: libc::c_int,
    bd: libc::c_int
  );
  fn av1_inv_txfm2d_add_32x32_c(
    input: *const i32,
    output: *mut u16,
    stride: libc::c_int,
    tx_type: libc::c_int,
    bd: libc::c_int
  );
}

pub fn forward_transform(
  input: &[i16], output: &mut [i32], stride: usize, tx_size: TxSize,
  tx_type: TxType, bit_depth: usize
) {
  match tx_size {
    TxSize::TX_4X4 => fht4x4(input, output, stride, tx_type, bit_depth),
    TxSize::TX_8X8 => fht8x8(input, output, stride, tx_type, bit_depth),
    TxSize::TX_16X16 => fht16x16(input, output, stride, tx_type, bit_depth),
    TxSize::TX_32X32 => fht32x32(input, output, stride, tx_type, bit_depth),
    _ => panic!("unimplemented tx size")
  }
}

pub fn inverse_transform_add(
  input: &[i32], output: &mut [u16], stride: usize, tx_size: TxSize,
  tx_type: TxType, bit_depth: usize
) {
  match tx_size {
    TxSize::TX_4X4 => iht4x4_add(input, output, stride, tx_type, bit_depth),
    TxSize::TX_8X8 => iht8x8_add(input, output, stride, tx_type, bit_depth),
    TxSize::TX_16X16 => iht16x16_add(input, output, stride, tx_type, bit_depth),
    TxSize::TX_32X32 => iht32x32_add(input, output, stride, tx_type, bit_depth),
    _ => panic!("unimplemented tx size")
  }
}

fn fht4x4(input: &[i16], output: &mut [i32], stride: usize, tx_type: TxType, bit_depth: usize) {
  unsafe {
    av1_fwd_txfm2d_4x4_c(
      input.as_ptr(),
      output.as_mut_ptr(),
      stride as libc::c_int,
      tx_type as libc::c_int,
      bit_depth as libc::c_int
    );
  }
}

fn iht4x4_add(
  input: &[i32], output: &mut [u16], stride: usize, tx_type: TxType, bit_depth: usize
) {
  // SIMD code may assert for transform types beyond TxType::IDTX.
  if tx_type < TxType::IDTX {
    if tx_type == TxType::DCT_DCT {
      inv_txfm2d_add_4x4_rs(input, output, stride, bit_depth);
    } else {
      unsafe {
        av1_inv_txfm2d_add_4x4(
          input.as_ptr(),
          output.as_mut_ptr(),
          stride as libc::c_int,
          tx_type as libc::c_int,
          bit_depth as libc::c_int
        );
      }
    }
  } else {
    unsafe {
      av1_inv_txfm2d_add_4x4_c(
        input.as_ptr(),
        output.as_mut_ptr(),
        stride as libc::c_int,
        tx_type as libc::c_int,
        bit_depth as libc::c_int
      );
    }
  }
}

fn fht8x8(input: &[i16], output: &mut [i32], stride: usize, tx_type: TxType, bit_depth: usize) {
  unsafe {
    av1_fwd_txfm2d_8x8_c(
      input.as_ptr(),
      output.as_mut_ptr(),
      stride as libc::c_int,
      tx_type as libc::c_int,
      bit_depth as libc::c_int
    );
  }
}

fn iht8x8_add(
  input: &[i32], output: &mut [u16], stride: usize, tx_type: TxType, bit_depth: usize
) {
  // SIMD code may assert for transform types beyond TxType::IDTX.
  if tx_type < TxType::IDTX {
    unsafe {
      av1_inv_txfm2d_add_8x8(
        input.as_ptr(),
        output.as_mut_ptr(),
        stride as libc::c_int,
        tx_type as libc::c_int,
        bit_depth as libc::c_int
      );
    }
  } else {
    unsafe {
      av1_inv_txfm2d_add_8x8_c(
        input.as_ptr(),
        output.as_mut_ptr(),
        stride as libc::c_int,
        tx_type as libc::c_int,
        bit_depth as libc::c_int
      );
    }
  }
}

fn fht16x16(
  input: &[i16], output: &mut [i32], stride: usize, tx_type: TxType, bit_depth: usize
) {
  unsafe {
    av1_fwd_txfm2d_16x16_c(
      input.as_ptr(),
      output.as_mut_ptr(),
      stride as libc::c_int,
      tx_type as libc::c_int,
      bit_depth as libc::c_int
    );
  }
}

fn iht16x16_add(
  input: &[i32], output: &mut [u16], stride: usize, tx_type: TxType, bit_depth: usize
) {
  unsafe {
    if tx_type < TxType::IDTX {
      // SSE C code asserts for transform types beyond TxType::IDTX.
      av1_inv_txfm2d_add_16x16(
        input.as_ptr(),
        output.as_mut_ptr(),
        stride as libc::c_int,
        tx_type as libc::c_int,
        bit_depth as libc::c_int
      );
    } else {
      av1_inv_txfm2d_add_16x16_c(
        input.as_ptr(),
        output.as_mut_ptr(),
        stride as libc::c_int,
        tx_type as libc::c_int,
        bit_depth as libc::c_int
      );
    }
  }
}

fn fht32x32(
  input: &[i16], output: &mut [i32], stride: usize, tx_type: TxType, bit_depth: usize
) {
  unsafe {
    av1_fwd_txfm2d_32x32_c(
      input.as_ptr(),
      output.as_mut_ptr(),
      stride as libc::c_int,
      tx_type as libc::c_int,
      bit_depth as libc::c_int
    );
  }
}

fn iht32x32_add(
  input: &[i32], output: &mut [u16], stride: usize, tx_type: TxType, bit_depth: usize
) {
  unsafe {
    if tx_type < TxType::IDTX {
      // SIMDI code may assert for transform types beyond TxType::IDTX.
      av1_inv_txfm2d_add_32x32(
        input.as_ptr(),
        output.as_mut_ptr(),
        stride as libc::c_int,
        tx_type as libc::c_int,
        bit_depth as libc::c_int
      );
    } else {
      av1_inv_txfm2d_add_32x32_c(
        input.as_ptr(),
        output.as_mut_ptr(),
        stride as libc::c_int,
        tx_type as libc::c_int,
        bit_depth as libc::c_int
      );
    }
  }
}
<|endoftext|>