<commit_before>// no-reformat

#[doc = "

    A demonstration module

    Contains documentation in various forms that rustdoc understands,
    for testing purposes. It doesn't surve any functional
    purpose. This here, for instance, is just some filler text.

    FIXME (1654): It would be nice if we could run some automated
    tests on this file

"];

#[doc = "The base price of a muffin on a non-holiday"]
const price_of_a_muffin: float = 70f;

type waitress = {
    hair_color: str
};

#[doc = "The type of things that produce omnomnom"]
enum omnomnomy {
    #[doc = "Delicious sugar cookies"]
    cookie,
    #[doc = "It's pizza"]
    pizza_pie([uint])
}

fn take_my_order_please(
    _waitress: waitress,
    _order: [omnomnomy]
) -> uint {

    #[doc(
        desc = "OMG would you take my order already?",
        args(_waitress = "The waitress that you want to bother",
             _order = "The order vector. It should be filled with food."),
        return = "The price of the order, including tax",
        failure = "This function is full of fail"
    )];

    fail;
}

fn take_my_order_now(
    _waitress: waitress,
    _order: [omnomnomy]
) -> uint {

    #[doc(
        desc = "OMG would you take my order already?",
        args(_waitress = "The waitress that you want to bother",
             _order = "The order vector. It should be filled with food."),
        return = "

        Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed
        nec molestie nisl. Duis massa risus, pharetra a scelerisque a,
        molestie eu velit. Donec mattis ligula at ante imperdiet ut
        dapibus mauris malesuada.

        Sed gravida nisi a metus elementum sit amet hendrerit dolor
        bibendum. Aenean sit amet neque massa, sed tempus tortor. Sed
        ut lobortis enim. Proin a mauris quis nunc fermentum ultrices
        eget a erat. Mauris in lectus vitae metus sodales
        auctor. Morbi nunc quam, ultricies at venenatis non,
        pellentesque ac dui.

        ",
        failure = "This function is full of fail"
    )];

    fail;
}

mod fortress_of_solitude {
    #[doc(
        brief = "Superman's vacation home",
        desc = "

        The fortress of solitude is located in the Arctic and it is
        cold. What you may not know about the fortress of solitude
        though is that it contains two separate bowling alleys. One of
        them features bumper-bowling and is kind of lame.

        Really, it's pretty cool.

    ")];

}

mod blade_runner {
    #[doc(
        brief = "Blade Runner is probably the best movie ever",
        desc = "I like that in the world of Blade Runner it is always
                raining, and that it's always night time. And Aliens
                was also a really good movie.

                Alien 3 was crap though."
    )];
}

#[doc(
    brief = "Bored",
    desc = "

    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed nec
    molestie nisl. Duis massa risus, pharetra a scelerisque a,
    molestie eu velit. Donec mattis ligula at ante imperdiet ut
    dapibus mauris malesuada. Sed gravida nisi a metus elementum sit
    amet hendrerit dolor bibendum. Aenean sit amet neque massa, sed
    tempus tortor. Sed ut lobortis enim. Proin a mauris quis nunc
    fermentum ultrices eget a erat. Mauris in lectus vitae metus
    sodales auctor. Morbi nunc quam, ultricies at venenatis non,
    pellentesque ac dui.

    Quisque vitae est id eros placerat laoreet sit amet eu
    nisi. Curabitur suscipit neque porttitor est euismod
    lacinia. Curabitur non quam vitae ipsum adipiscing
    condimentum. Mauris ut ante eget metus sollicitudin
    blandit. Aliquam erat volutpat. Morbi sed nisl mauris. Nulla
    facilisi. Phasellus at mollis ipsum. Maecenas sed convallis
    sapien. Nullam in ligula turpis. Pellentesque a neque augue. Sed
    eget ante feugiat tortor congue auctor ac quis ante. Proin
    condimentum lacinia tincidunt.

")]
resource bored(bored: bool) {
    log(error, bored);
}

#[doc(
    brief = "The Shunned House",
    desc = "

From even the greatest of horrors irony is seldom absent. Sometimes it
enters directly into the composition of the events, while sometimes it
relates only to their fortuitous position among persons and
places. The latter sort is splendidly exemplified by a case in the
ancient city of Providence, where in the late forties Edgar Allan Poe
used to sojourn often during his unsuccessful wooing of the gifted
poetess, Mrs.  Whitman. Poe generally stopped at the Mansion House in
Benefit Street--the renamed Golden Ball Inn whose roof has sheltered
Washington, Jefferson, and Lafayette--and his favorite walk led
northward along the same street to Mrs. Whitman's home and the
neighboring hillside churchyard of St. John's, whose hidden expanse of
Eighteenth Century gravestones had for him a peculiar fascination.

")]
iface the_shunned_house {
    #[doc(
        desc = "

    Now the irony is this. In this walk, so many times repeated, the
    world's greatest master of the terrible and the bizarre was
    obliged to pass a particular house on the eastern side of the
    street; a dingy, antiquated structure perched on the abruptly
    rising side hill, with a great unkempt yard dating from a time
    when the region was partly open country. It does not appear that
    he ever wrote or spoke of it, nor is there any evidence that he
    even noticed it. And yet that house, to the two persons in
    possession of certain information, equals or outranks in horror
    the wildest fantasy of the genius who so often passed it
    unknowingly, and stands starkly leering as a symbol of all that is
    unutterably hideous.

    ",
        args(
            a =
            "A yard dating from a time when the region was partly
             open country"
    ))]
    fn dingy_house(unkempt_yard: int);

    #[doc(
        desc = "

    The house was--and for that matter still is--of a kind to attract
    the attention of the curious. Originally a farm or semi-farm
    building, it followed the average New England colonial lines of
    the middle Eighteenth Century--the prosperous peaked-roof sort,
    with two stories and dormerless attic, and with the Georgian
    doorway and interior panelling dictated by the progress of taste
    at that time. It faced south, with one gable end buried to the
    lower windows in the eastward rising hill, and the other exposed
    to the foundations toward the street. Its construction, over a
    century and a half ago, had followed the grading and straightening
    of the road in that especial vicinity; for Benefit Street--at
    first called Back Street--was laid out as a lane winding amongst
    the graveyards of the first settlers, and straightened only when
    the removal of the bodies to the North Burial Ground made it
    decently possible to cut through the old family plots.

    ",
        return = "A dingy house with an unkempt yard",
        failure = "Will fail if bodies are removed from premises"
    )]
    fn construct() -> bool;
}

#[doc = "Whatever"]
impl of the_shunned_house for omnomnomy {
    #[doc(args(_unkempt_yard = "Whatever"))]
    fn dingy_house(_unkempt_yard: int) {
    }

    fn construct() -> bool {
        fail;
    }
}<commit_msg>rustdoc: Update the demo module<commit_after>// no-reformat

#[doc = "

    A demonstration module

    Contains documentation in various forms that rustdoc understands,
    for testing purposes. It doesn't surve any functional
    purpose. This here, for instance, is just some filler text.

    FIXME (1654): It would be nice if we could run some automated
    tests on this file

"];

#[doc = "The base price of a muffin on a non-holiday"]
const price_of_a_muffin: float = 70f;

type waitress = {
    hair_color: str
};

#[doc = "The type of things that produce omnomnom"]
enum omnomnomy {
    #[doc = "Delicious sugar cookies"]
    cookie,
    #[doc = "It's pizza"]
    pizza_pie([uint])
}

fn take_my_order_please(
    _waitress: waitress,
    _order: [omnomnomy]
) -> uint {

    #[doc = "
    OMG would you take my order already?

    # Arguments

    * _waitress - The waitress that you want to bother
    * _order - The order vector. It should be filled with food

    # Return

    The price of the order, including tax

    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed nec
    molestie nisl. Duis massa risus, pharetra a scelerisque a,
    molestie eu velit. Donec mattis ligula at ante imperdiet ut
    dapibus mauris malesuada.

    Sed gravida nisi a metus elementum sit amet hendrerit dolor
    bibendum. Aenean sit amet neque massa, sed tempus tortor. Sed ut
    lobortis enim. Proin a mauris quis nunc fermentum ultrices eget a
    erat. Mauris in lectus vitae metus sodales auctor. Morbi nunc
    quam, ultricies at venenatis non, pellentesque ac dui.

    # Failure

    This function is full of fail
    "];

    fail;
}

mod fortress_of_solitude {
    #[doc = "
    Superman's vacation home

    The fortress of solitude is located in the Arctic and it is
    cold. What you may not know about the fortress of solitude
    though is that it contains two separate bowling alleys. One of
    them features bumper-bowling and is kind of lame.

    Really, it's pretty cool.

    "];

}

mod blade_runner {
    #[doc = "
    Blade Runner is probably the best movie ever

    I like that in the world of Blade Runner it is always
    raining, and that it's always night time. And Aliens
    was also a really good movie.

    Alien 3 was crap though.
    "];
}

#[doc = "
Bored

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed nec
molestie nisl. Duis massa risus, pharetra a scelerisque a,
molestie eu velit. Donec mattis ligula at ante imperdiet ut
dapibus mauris malesuada. Sed gravida nisi a metus elementum sit
amet hendrerit dolor bibendum. Aenean sit amet neque massa, sed
tempus tortor. Sed ut lobortis enim. Proin a mauris quis nunc
fermentum ultrices eget a erat. Mauris in lectus vitae metus
sodales auctor. Morbi nunc quam, ultricies at venenatis non,
pellentesque ac dui.

Quisque vitae est id eros placerat laoreet sit amet eu
nisi. Curabitur suscipit neque porttitor est euismod
lacinia. Curabitur non quam vitae ipsum adipiscing
condimentum. Mauris ut ante eget metus sollicitudin
blandit. Aliquam erat volutpat. Morbi sed nisl mauris. Nulla
facilisi. Phasellus at mollis ipsum. Maecenas sed convallis
sapien. Nullam in ligula turpis. Pellentesque a neque augue. Sed
eget ante feugiat tortor congue auctor ac quis ante. Proin
condimentum lacinia tincidunt.

"]
resource bored(bored: bool) {
    log(error, bored);
}

#[doc = "
The Shunned House

From even the greatest of horrors irony is seldom absent. Sometimes it
enters directly into the composition of the events, while sometimes it
relates only to their fortuitous position among persons and
places. The latter sort is splendidly exemplified by a case in the
ancient city of Providence, where in the late forties Edgar Allan Poe
used to sojourn often during his unsuccessful wooing of the gifted
poetess, Mrs.  Whitman. Poe generally stopped at the Mansion House in
Benefit Street--the renamed Golden Ball Inn whose roof has sheltered
Washington, Jefferson, and Lafayette--and his favorite walk led
northward along the same street to Mrs. Whitman's home and the
neighboring hillside churchyard of St. John's, whose hidden expanse of
Eighteenth Century gravestones had for him a peculiar fascination.

"]
iface the_shunned_house {
    #[doc = "
    Now the irony is this. In this walk, so many times repeated, the
    world's greatest master of the terrible and the bizarre was
    obliged to pass a particular house on the eastern side of the
    street; a dingy, antiquated structure perched on the abruptly
    rising side hill, with a great unkempt yard dating from a time
    when the region was partly open country. It does not appear that
    he ever wrote or spoke of it, nor is there any evidence that he
    even noticed it. And yet that house, to the two persons in
    possession of certain information, equals or outranks in horror
    the wildest fantasy of the genius who so often passed it
    unknowingly, and stands starkly leering as a symbol of all that is
    unutterably hideous.

    # Arguments

    * unkempt_yard - A yard dating from a time when the region was partly
                     open country
    "]
    fn dingy_house(unkempt_yard: int);

    #[doc = "
    The house was--and for that matter still is--of a kind to attract
    the attention of the curious. Originally a farm or semi-farm
    building, it followed the average New England colonial lines of
    the middle Eighteenth Century--the prosperous peaked-roof sort,
    with two stories and dormerless attic, and with the Georgian
    doorway and interior panelling dictated by the progress of taste
    at that time. It faced south, with one gable end buried to the
    lower windows in the eastward rising hill, and the other exposed
    to the foundations toward the street. Its construction, over a
    century and a half ago, had followed the grading and straightening
    of the road in that especial vicinity; for Benefit Street--at
    first called Back Street--was laid out as a lane winding amongst
    the graveyards of the first settlers, and straightened only when
    the removal of the bodies to the North Burial Ground made it
    decently possible to cut through the old family plots.
    "]
    fn construct() -> bool;
}

#[doc = "Whatever"]
impl of the_shunned_house for omnomnomy {
    fn dingy_house(_unkempt_yard: int) {
    }

    fn construct() -> bool {
        fail;
    }
}<|endoftext|>


<commit_before>#![doc="Statistical moments
"]

// local imports
use entry::Zero;
use number::Number;
use matrix::matrix::Matrix;
use matrix::traits::{Shape, MatrixBuffer, Strided,
 StridedNumberMatrix};

/// Computes sum over columns and returns a row vector
pub fn sum_cw<T:Number>(m : & StridedNumberMatrix<T>) -> Matrix<T> {
    let cols = m.num_cols();
    let rows = m.num_rows();
    let mut result = Matrix::new(1, cols);
    let ptr = m.as_ptr();
    let stride = m.stride() as int;
    let mut offset = 0i;
    for c in range(0, cols) {
        let mut sum : T = Zero::zero();
        for r in range(0, rows){
            sum = sum + unsafe{*ptr.offset(offset + r as int)};
        }
        offset += stride;
        result.set(0, c, sum);
    }
    result
}

/// Computes sum over rows and returns a column vector
pub fn sum_rw<T:Number>(m : & Matrix<T>) -> Matrix<T> {
    let cols = m.num_cols();
    let rows = m.num_rows();
    let mut result = Matrix::new(rows, 1);
    let ptr = m.as_ptr();
    let stride = m.stride() as int;
    let mut offset = 0i;
    for r in range(0, rows) {
        let mut sum : T = Zero::zero();
        let mut src_offset  = offset;
        for _ in range(0, cols){
            sum = sum + unsafe{*ptr.offset(src_offset)};
            src_offset += stride;
        }
        offset += 1;
        result.set(r, 0, sum);
    }
    result
}

/// Computes sum over columns and returns a row vector
pub fn sum_sqr_cw<T:Number>(m : & Matrix<T>) -> Matrix<T> {
    let cols = m.num_cols();
    let rows = m.num_rows();
    let mut result = Matrix::new(1, cols);
    let ptr = m.as_ptr();
    let stride = m.stride() as int;
    let mut offset = 0i;
    for c in range(0, cols) {
        let mut sum : T = Zero::zero();
        for r in range(0, rows){
            let v  = unsafe{*ptr.offset(offset + r as int)};
            sum = sum + v * v;
        }
        offset += stride;
        result.set(0, c, sum);
    }
    result
}

/// Computes sum over rows and returns a column vector
pub fn sum_sqr_rw<T:Number>(m : & Matrix<T>) -> Matrix<T> {
    let cols = m.num_cols();
    let rows = m.num_rows();
    let mut result = Matrix::new(rows, 1);
    let ptr = m.as_ptr();
    let stride = m.stride() as int;
    let mut offset = 0i;
    for r in range(0, rows) {
        let mut sum : T = Zero::zero();
        let mut src_offset  = offset;
        for _ in range(0, cols){
            let v = unsafe{*ptr.offset(src_offset)};
            sum = sum + v * v;
            src_offset += stride;
        }
        offset += 1;
        result.set(r, 0, sum);
    }
    result
}

/******************************************************
 *
 *   Unit tests
 *
 *******************************************************/
#[cfg(test)]
mod test{

    use super::*;
    use api::*;

    #[test]
    fn test_moment_sum_cw_1(){
        let m = matrix_rw_i32(3, 3, &[
            1, 2, 3,
            4, 5, 6,
            7, 8, 9]);
        let s = sum_cw(&m);
        assert_eq!(s, matrix_cw_i32(1,3, &[12, 15, 18]));
    }

    #[test]
    fn test_moment_sum_rw_1(){
        let m = matrix_rw_i32(3, 3, &[
            1, 2, 3,
            4, 5, 6,
            7, 8, 9]);
        let s = sum_rw(&m);
        assert_eq!(s, matrix_rw_i32(3,1, &[6, 15, 24]));
    }

    #[test]
    fn test_moment_sum_sqr_cw_1(){
        let m = matrix_rw_i32(3, 3, &[
            1, 1, 2,
            2, 2, 1,
            3, 2, 2]);
        let s = sum_sqr_cw(&m);
        assert_eq!(s, matrix_cw_i32(1,3, &[14, 9, 9]));
    }

    #[test]
    fn test_moment_sum_sqr_rw_1(){
        let m = matrix_rw_i32(3, 3, &[
            1, 1, 2,
            2, 2, 1,
            3, 2, 2]);
        let s = sum_sqr_rw(&m);
        assert_eq!(s, matrix_cw_i32(3,1, &[6, 9, 17]));
    }

}<commit_msg>Made methods work with any StridedNumberMatrix<commit_after>#![doc="Statistical moments
"]

// local imports
use entry::Zero;
use number::Number;
use matrix::matrix::Matrix;
use matrix::traits::{Shape, MatrixBuffer, Strided,
 StridedNumberMatrix};

/// Computes sum over columns and returns a row vector
pub fn sum_cw<T:Number>(m : & StridedNumberMatrix<T>) -> Matrix<T> {
    let cols = m.num_cols();
    let rows = m.num_rows();
    let mut result = Matrix::new(1, cols);
    let ptr = m.as_ptr();
    let stride = m.stride() as int;
    let mut offset = m.start_offset();
    for c in range(0, cols) {
        let mut sum : T = Zero::zero();
        for r in range(0, rows){
            sum = sum + unsafe{*ptr.offset(offset + r as int)};
        }
        offset += stride;
        result.set(0, c, sum);
    }
    result
}

/// Computes sum over rows and returns a column vector
pub fn sum_rw<T:Number>(m : & StridedNumberMatrix<T>) -> Matrix<T> {
    let cols = m.num_cols();
    let rows = m.num_rows();
    let mut result = Matrix::new(rows, 1);
    let ptr = m.as_ptr();
    let stride = m.stride() as int;
    let mut offset = m.start_offset();
    for r in range(0, rows) {
        let mut sum : T = Zero::zero();
        let mut src_offset  = offset;
        for _ in range(0, cols){
            sum = sum + unsafe{*ptr.offset(src_offset)};
            src_offset += stride;
        }
        offset += 1;
        result.set(r, 0, sum);
    }
    result
}

/// Computes sum over columns and returns a row vector
pub fn sum_sqr_cw<T:Number>(m : & StridedNumberMatrix<T>) -> Matrix<T> {
    let cols = m.num_cols();
    let rows = m.num_rows();
    let mut result = Matrix::new(1, cols);
    let ptr = m.as_ptr();
    let stride = m.stride() as int;
    let mut offset = m.start_offset();
    for c in range(0, cols) {
        let mut sum : T = Zero::zero();
        for r in range(0, rows){
            let v  = unsafe{*ptr.offset(offset + r as int)};
            sum = sum + v * v;
        }
        offset += stride;
        result.set(0, c, sum);
    }
    result
}

/// Computes sum over rows and returns a column vector
pub fn sum_sqr_rw<T:Number>(m : & StridedNumberMatrix<T>) -> Matrix<T> {
    let cols = m.num_cols();
    let rows = m.num_rows();
    let mut result = Matrix::new(rows, 1);
    let ptr = m.as_ptr();
    let stride = m.stride() as int;
    let mut offset = m.start_offset();
    for r in range(0, rows) {
        let mut sum : T = Zero::zero();
        let mut src_offset  = offset;
        for _ in range(0, cols){
            let v = unsafe{*ptr.offset(src_offset)};
            sum = sum + v * v;
            src_offset += stride;
        }
        offset += 1;
        result.set(r, 0, sum);
    }
    result
}

/******************************************************
 *
 *   Unit tests
 *
 *******************************************************/
#[cfg(test)]
mod test{

    use super::*;
    use api::*;

    #[test]
    fn test_moment_sum_cw_1(){
        let m = matrix_rw_i32(3, 3, &[
            1, 2, 3,
            4, 5, 6,
            7, 8, 9]);
        let s = sum_cw(&m);
        assert_eq!(s, matrix_cw_i32(1,3, &[12, 15, 18]));
    }

    #[test]
    fn test_moment_sum_cw_2(){
        let m = matrix_rw_i32(3, 3, &[
            1, 2, 3,
            4, 5, 6,
            7, 8, 9]);
        let v = m.view(1, 1, 2, 2);
        let s = sum_cw(&v);
        assert_eq!(s, matrix_cw_i32(1,2, &[13, 15]));
    }

    #[test]
    fn test_moment_sum_rw_1(){
        let m = matrix_rw_i32(3, 3, &[
            1, 2, 3,
            4, 5, 6,
            7, 8, 9]);
        let s = sum_rw(&m);
        assert_eq!(s, matrix_rw_i32(3,1, &[6, 15, 24]));
    }

    #[test]
    fn test_moment_sum_rw_2(){
        let m = matrix_rw_i32(3, 3, &[
            1, 2, 3,
            4, 5, 6,
            7, 8, 9]);
        let v = m.view(1, 1, 2, 2);
        let s = sum_rw(&v);
        assert_eq!(s, matrix_rw_i32(2,1, &[11, 17]));
    }

    #[test]
    fn test_moment_sum_sqr_cw_1(){
        let m = matrix_rw_i32(3, 3, &[
            1, 1, 2,
            2, 2, 1,
            3, 2, 2]);
        let s = sum_sqr_cw(&m);
        assert_eq!(s, matrix_cw_i32(1,3, &[14, 9, 9]));
    }

    #[test]
    fn test_moment_sum_sqr_cw_2(){
        let m = matrix_rw_i32(3, 3, &[
            1, 1, 2,
            2, 2, 1,
            3, 2, 2]);
        let v = m.view(1, 1, 2, 2);
        let s = sum_sqr_cw(&v);
        assert_eq!(s, matrix_cw_i32(1,2, &[8, 5]));
    }

    #[test]
    fn test_moment_sum_sqr_rw_1(){
        let m = matrix_rw_i32(3, 3, &[
            1, 1, 2,
            2, 2, 1,
            3, 2, 2]);
        let s = sum_sqr_rw(&m);
        assert_eq!(s, matrix_cw_i32(3,1, &[6, 9, 17]));
    }

    #[test]
    fn test_moment_sum_sqr_rw_2(){
        let m = matrix_rw_i32(3, 3, &[
            1, 1, 2,
            2, 2, 1,
            3, 2, 2]);
        let v = m.view(1, 1, 2, 2);
        let s = sum_sqr_rw(&v);
        assert_eq!(s, matrix_cw_i32(2,1, &[5, 8]));
    }
}<|endoftext|>


<commit_before>use super::column::Column;
use super::table::Table;
use super::Error;
use std::io::{Write, Read, Seek, SeekFrom};

/// The Rows struct represents the file level implementation
/// of the table
pub struct Rows<B: Write + Read + Seek> {
    buf: B,

    table_header_offset: u32,
    columns_offset: u32,
}

// TODO: implement insert_row and select_row
impl<B: Write + Read + Seek> Rows<B> {
    pub fn new(buf: B, table_header_offset: u32, columns_offset: u32) -> Self {
        Rows {
            buf: buf,
            table_header_offset: table_header_offset,
            columns_offset: columns_offset
        }
    }

    pub fn insert_row(&mut self, data: &[u8]) -> Result<usize, Error> {
        info!("inserting row");
        try!(self.buf.seek(SeekFrom::End(0)));
        self.write_bytes(data)
    }

    fn write_bytes(&mut self, data: &[u8]) -> Result<usize, Error> {
        match self.buf.write_all(data) {
            Ok(_) => Ok(data.len()),
            Err(e) => Err(Error::Io(e))
        }
    }
}
<commit_msg>added TODO to write select_bytes for rows<commit_after>use super::column::Column;
use super::table::Table;
use super::Error;
use std::io::{Write, Read, Seek, SeekFrom};

/// The Rows struct represents the file level implementation
/// of the table
pub struct Rows<B: Write + Read + Seek> {
    buf: B,

    table_header_offset: u32,
    columns_offset: u32,
}

impl<B: Write + Read + Seek> Rows<B> {
    pub fn new(buf: B, table_header_offset: u32, columns_offset: u32) -> Self {
        Rows {
            buf: buf,
            table_header_offset: table_header_offset,
            columns_offset: columns_offset
        }
    }

    pub fn insert_data(&mut self, data: &[u8]) -> Result<usize, Error> {
        info!("inserting row");
        try!(self.buf.seek(SeekFrom::End(0)));
        self.write_bytes(data)
    }

    fn write_bytes(&mut self, data: &[u8]) -> Result<usize, Error> {
        match self.buf.write_all(data) {
            Ok(_) => Ok(data.len()),
            Err(e) => Err(Error::Io(e))
        }
    }

    // TODO: write select_data function
}

<|endoftext|>


<commit_before>#![feature(try_trait)]

extern crate pgn_reader;
extern crate memmap;
extern crate madvise;
extern crate btoi;
extern crate rand;
extern crate reqwest;

use std::env;
use std::mem;
use std::cmp::min;
use std::fs::File;
use std::option::NoneError;
use std::io::Read;

use memmap::Mmap;
use madvise::{AccessPattern, AdviseMemory};
use pgn_reader::{Visitor, Skip, Reader, San};
use btoi::ParseIntegerError;
use rand::{random, Closed01};

const BATCH_SIZE: usize = 1;

const MAX_PLIES: usize = 50;

#[derive(Debug)]
enum TimeControl {
    UltraBullet,
    Bullet,
    Blitz,
    Classical,
    Correspondence,
}

#[derive(Debug)]
struct TimeControlError;

impl From<NoneError> for TimeControlError {
    fn from(_: NoneError) -> TimeControlError {
        TimeControlError { }
    }
}

impl From<ParseIntegerError> for TimeControlError {
    fn from(_: ParseIntegerError) -> TimeControlError {
        TimeControlError { }
    }
}

impl TimeControl {
    fn from_seconds_and_increment(seconds: u64, increment: u64) -> TimeControl {
        let total = seconds + 40 * increment;

        if total < 30 {
            TimeControl::UltraBullet
        } else if total < 180 {
            TimeControl::Bullet
        } else if total < 480 {
            TimeControl::Blitz
        } else if total < 21600 {
            TimeControl::Classical
        } else {
            TimeControl::Correspondence
        }
    }

    fn from_bytes(bytes: &[u8]) -> Result<TimeControl, TimeControlError> {
        if bytes == b"-" {
            return Ok(TimeControl::Correspondence);
        }

        let mut parts = bytes.splitn(2, |ch| *ch == b'+');
        let seconds = btoi::btou(parts.next()?)?;
        let increment = btoi::btou(parts.next()?)?;
        Ok(TimeControl::from_seconds_and_increment(seconds, increment))
    }
}

struct Indexer {
    white_elo: i16,
    black_elo: i16,
    time_control: TimeControl,
    skip: bool,

    current_game: Vec<u8>,
    plies: usize,

    batch: Vec<u8>,
    batch_size: usize,
}

impl Indexer {
    fn new() -> Indexer {
        Indexer {
            white_elo: 0,
            black_elo: 0,
            time_control: TimeControl::Correspondence,
            skip: true,

            current_game: Vec::new(),
            plies: 0,

            batch: Vec::new(),
            batch_size: 0,
        }
    }

    fn send(&mut self) {
        if self.batch_size > 0 {
            self.batch_size = 0;

            let mut res = reqwest::Client::new()
                .put("http://localhost:9000/import/lichess")
                .body(mem::replace(&mut self.batch, Vec::new()))
                .send().expect("send batch");

            let mut answer = String::new();
            res.read_to_string(&mut answer).expect("decode response");
            println!("{}", answer);
            assert!(res.status().is_success());
        }
    }
}

impl<'pgn> Visitor<'pgn> for Indexer {
    type Result = ();

    fn begin_game(&mut self) {
        self.current_game.clear();
        self.plies = 0;
    }

    fn begin_headers(&mut self) {
        self.white_elo = 0;
        self.black_elo = 0;
        self.time_control = TimeControl::Correspondence;
    }

    fn header(&mut self, key: &'pgn [u8], value: &'pgn [u8]) {
        if key == b"WhiteElo" {
            self.white_elo = if value == b"?" { 0 } else { btoi::btoi(value).expect("WhiteElo") };
        } else if key == b"BlackElo" {
            self.black_elo = if value == b"?" { 0 } else { btoi::btoi(value).expect("BlackElo") };
        } else if key == b"TimeControl" {
            self.time_control = TimeControl::from_bytes(value).expect("TimeControl");
        } else if key == b"Variant" {
            assert_eq!(value, b"Standard");
            return; // we add this unconditionally later
        }

        let (key, value) = if key == b"Site" {
            (&b"LichessID"[..], value.rsplitn(2, |ch| *ch == b'/').next().expect("Site"))
        } else {
            (key, value)
        };

        self.current_game.push(b'[');
        self.current_game.extend(key);
        self.current_game.extend(b" \"");
        self.current_game.extend(value);
        self.current_game.extend(b"\"]\n");
    }

    fn end_headers(&mut self) -> Skip {
        self.current_game.extend(b"[Variant \"Standard\"]\n");
        self.current_game.push(b'\n');

        let rating = (self.white_elo + self.black_elo) / 2;

        let probability = match self.time_control {
            TimeControl::Correspondence => 1.0,
            TimeControl::Classical if rating >= 2000 => 1.0,
            TimeControl::Classical if rating >= 1800 => 2.0 / 5.0,
            TimeControl::Classical => 1.0 / 8.0,
            TimeControl::Blitz if rating >= 2000 => 1.0,
            TimeControl::Blitz if rating >= 1800 => 1.0 / 4.0,
            TimeControl::Blitz => 1.0 / 15.0,
            TimeControl::Bullet if rating >= 2300 => 1.0,
            TimeControl::Bullet if rating >= 2200 => 4.0 / 5.0,
            TimeControl::Bullet if rating >= 2000 => 1.0 / 4.0,
            TimeControl::Bullet if rating >= 1800 => 1.0 / 7.0,
            _ => 1.0 / 20.0,
        };

        let Closed01(rnd) = random::<Closed01<f64>>();
        let accept = min(self.white_elo, self.black_elo) >= 1500 && probability >= rnd;

        self.skip = !accept;
        Skip(self.skip)
    }

    fn san(&mut self, san: San) {
        if self.plies < MAX_PLIES {
            if self.plies > 0 {
                self.current_game.push(b' ');
            }

            self.current_game.extend(san.to_string().as_bytes());
            self.plies += 1;
        }
    }

    fn begin_variation(&mut self) -> Skip {
        Skip(true) // stay in the mainline
    }

    fn end_game(&mut self, _game: &'pgn [u8]) {
        if !self.skip && self.plies > 8 {
            if self.batch_size >= BATCH_SIZE {
                self.send();
            }

            if self.batch_size > 0 {
                self.batch.extend(b"\n\n");
            }

            self.batch.extend(&self.current_game);
            self.batch_size += 1;
        }
    }
}

fn main() {
    for arg in env::args().skip(1) {
        eprintln!("% indexing {} ...", arg);
        let file = File::open(&arg).expect("fopen");
        let pgn = unsafe { Mmap::map(&file).expect("mmap") };
        pgn.advise_memory_access(AccessPattern::Sequential).expect("madvise");

        let mut indexer = Indexer::new();
        Reader::new(&mut indexer, &pgn[..]).read_all();
        indexer.send(); // send last
    }
}
<commit_msg>fix import batching<commit_after>#![feature(try_trait)]

extern crate pgn_reader;
extern crate memmap;
extern crate madvise;
extern crate btoi;
extern crate rand;
extern crate reqwest;

use std::env;
use std::mem;
use std::cmp::min;
use std::fs::File;
use std::option::NoneError;
use std::io::Read;

use memmap::Mmap;
use madvise::{AccessPattern, AdviseMemory};
use pgn_reader::{Visitor, Skip, Reader, San};
use btoi::ParseIntegerError;
use rand::{random, Closed01};

const BATCH_SIZE: usize = 50;

const MAX_PLIES: usize = 50;

#[derive(Debug)]
enum TimeControl {
    UltraBullet,
    Bullet,
    Blitz,
    Classical,
    Correspondence,
}

#[derive(Debug)]
struct TimeControlError;

impl From<NoneError> for TimeControlError {
    fn from(_: NoneError) -> TimeControlError {
        TimeControlError { }
    }
}

impl From<ParseIntegerError> for TimeControlError {
    fn from(_: ParseIntegerError) -> TimeControlError {
        TimeControlError { }
    }
}

impl TimeControl {
    fn from_seconds_and_increment(seconds: u64, increment: u64) -> TimeControl {
        let total = seconds + 40 * increment;

        if total < 30 {
            TimeControl::UltraBullet
        } else if total < 180 {
            TimeControl::Bullet
        } else if total < 480 {
            TimeControl::Blitz
        } else if total < 21600 {
            TimeControl::Classical
        } else {
            TimeControl::Correspondence
        }
    }

    fn from_bytes(bytes: &[u8]) -> Result<TimeControl, TimeControlError> {
        if bytes == b"-" {
            return Ok(TimeControl::Correspondence);
        }

        let mut parts = bytes.splitn(2, |ch| *ch == b'+');
        let seconds = btoi::btou(parts.next()?)?;
        let increment = btoi::btou(parts.next()?)?;
        Ok(TimeControl::from_seconds_and_increment(seconds, increment))
    }
}

struct Indexer {
    white_elo: i16,
    black_elo: i16,
    time_control: TimeControl,
    skip: bool,

    current_game: Vec<u8>,
    plies: usize,

    batch: Vec<u8>,
    batch_size: usize,
}

impl Indexer {
    fn new() -> Indexer {
        Indexer {
            white_elo: 0,
            black_elo: 0,
            time_control: TimeControl::Correspondence,
            skip: true,

            current_game: Vec::new(),
            plies: 0,

            batch: Vec::new(),
            batch_size: 0,
        }
    }

    fn send(&mut self) {
        if self.batch_size > 0 {
            self.batch_size = 0;

            let mut res = reqwest::Client::new()
                .put("http://localhost:9000/import/lichess")
                .body(mem::replace(&mut self.batch, Vec::new()))
                .send().expect("send batch");

            let mut answer = String::new();
            res.read_to_string(&mut answer).expect("decode response");
            println!("{}", answer);
            assert!(res.status().is_success());
        }
    }
}

impl<'pgn> Visitor<'pgn> for Indexer {
    type Result = ();

    fn begin_game(&mut self) {
        self.current_game.clear();
        self.plies = 0;
    }

    fn begin_headers(&mut self) {
        self.white_elo = 0;
        self.black_elo = 0;
        self.time_control = TimeControl::Correspondence;
    }

    fn header(&mut self, key: &'pgn [u8], value: &'pgn [u8]) {
        if key == b"WhiteElo" {
            self.white_elo = if value == b"?" { 0 } else { btoi::btoi(value).expect("WhiteElo") };
        } else if key == b"BlackElo" {
            self.black_elo = if value == b"?" { 0 } else { btoi::btoi(value).expect("BlackElo") };
        } else if key == b"TimeControl" {
            self.time_control = TimeControl::from_bytes(value).expect("TimeControl");
        } else if key == b"Variant" {
            assert_eq!(value, b"Standard");
            return; // we add this unconditionally later
        }

        let (key, value) = if key == b"Site" {
            (&b"LichessID"[..], value.rsplitn(2, |ch| *ch == b'/').next().expect("Site"))
        } else {
            (key, value)
        };

        self.current_game.push(b'[');
        self.current_game.extend(key);
        self.current_game.extend(b" \"");
        self.current_game.extend(value);
        self.current_game.extend(b"\"]\n");
    }

    fn end_headers(&mut self) -> Skip {
        self.current_game.extend(b"[Variant \"Standard\"]\n");
        self.current_game.push(b'\n');

        let rating = (self.white_elo + self.black_elo) / 2;

        let probability = match self.time_control {
            TimeControl::Correspondence => 1.0,
            TimeControl::Classical if rating >= 2000 => 1.0,
            TimeControl::Classical if rating >= 1800 => 2.0 / 5.0,
            TimeControl::Classical => 1.0 / 8.0,
            TimeControl::Blitz if rating >= 2000 => 1.0,
            TimeControl::Blitz if rating >= 1800 => 1.0 / 4.0,
            TimeControl::Blitz => 1.0 / 15.0,
            TimeControl::Bullet if rating >= 2300 => 1.0,
            TimeControl::Bullet if rating >= 2200 => 4.0 / 5.0,
            TimeControl::Bullet if rating >= 2000 => 1.0 / 4.0,
            TimeControl::Bullet if rating >= 1800 => 1.0 / 7.0,
            _ => 1.0 / 20.0,
        };

        let Closed01(rnd) = random::<Closed01<f64>>();
        let accept = min(self.white_elo, self.black_elo) >= 1500 && probability >= rnd;

        self.skip = !accept;
        Skip(self.skip)
    }

    fn san(&mut self, san: San) {
        if self.plies < MAX_PLIES {
            if self.plies > 0 {
                self.current_game.push(b' ');
            }

            self.current_game.extend(san.to_string().as_bytes());
            self.plies += 1;
        }
    }

    fn begin_variation(&mut self) -> Skip {
        Skip(true) // stay in the mainline
    }

    fn end_game(&mut self, _game: &'pgn [u8]) {
        if !self.skip && self.plies > 8 {
            if self.batch_size >= BATCH_SIZE {
                self.send();
            }

            if self.batch_size > 0 {
                self.batch.extend(b"\n\n\n");
            }

            self.batch.extend(&self.current_game);
            self.batch_size += 1;
        }
    }
}

fn main() {
    for arg in env::args().skip(1) {
        eprintln!("% indexing {} ...", arg);
        let file = File::open(&arg).expect("fopen");
        let pgn = unsafe { Mmap::map(&file).expect("mmap") };
        pgn.advise_memory_access(AccessPattern::Sequential).expect("madvise");

        let mut indexer = Indexer::new();
        Reader::new(&mut indexer, &pgn[..]).read_all();
        indexer.send(); // send last
    }
}
<|endoftext|>


<commit_before>use crate::builtins::Blob;
use crate::instructions::Expression;
use crate::sections::CustomSection;
use crate::sections::Func;
use crate::sections::{ExportDesc, ImportDesc, ImportPath, Section};
use crate::types::FuncType;
use crate::types::GlobalType;
use crate::types::MemType;
use crate::types::TableType;
use custom_debug::CustomDebug;
use std::cell::{Ref, RefCell, RefMut};
use std::convert::TryFrom;
use std::iter::{Extend, FromIterator};
use std::rc::Rc;

fn to_item<T>(value: T) -> Option<Rc<RefCell<T>>> {
    Some(Rc::new(RefCell::new(value)))
}

pub struct IndexedCollection<T> {
    items: Vec<Option<Rc<RefCell<T>>>>,
    deleted: Vec<u32>,
}

impl<T> Default for IndexedCollection<T> {
    fn default() -> Self {
        Self {
            items: Vec::new(),
            deleted: Vec::new(),
        }
    }
}

impl<T> FromIterator<T> for IndexedCollection<T> {
    fn from_iter<I: IntoIterator<Item = T>>(iter: I) -> Self {
        IndexedCollection {
            items: iter.into_iter().map(to_item).collect(),
            deleted: Vec::new(),
        }
    }
}

impl<T> IndexedCollection<T> {
    fn get_rc_refcell(&self, index: u32) -> Option<&Rc<RefCell<T>>> {
        self.items.get(usize::try_from(index).unwrap())?.as_ref()
    }

    fn get_refcell(&self, index: u32) -> Option<&RefCell<T>> {
        self.get_rc_refcell(index).map(|r| &**r)
    }

    pub fn get(&self, index: u32) -> Option<Ref<T>> {
        self.get_refcell(index).map(RefCell::borrow)
    }

    pub fn get_mut(&self, index: u32) -> Option<RefMut<T>> {
        self.get_refcell(index).map(RefCell::borrow_mut)
    }

    pub fn push(&mut self, value: T) -> u32 {
        match self.deleted.pop() {
            Some(index) => {
                self.items[usize::try_from(index).unwrap()] = to_item(value);
                index
            }
            None => {
                let index = u32::try_from(self.items.len()).unwrap();
                self.items.push(to_item(value));
                index
            }
        }
    }

    // Don't allow removals for referenced elements.
    // Will return a number of references as error (excluding the owner).
    pub fn try_remove(&mut self, index: u32) -> Option<Result<T, usize>> {
        let place = self.items.get_mut(usize::try_from(index).unwrap())?;
        Some(match Rc::try_unwrap(place.take()?) {
            Ok(value) => {
                self.deleted.push(index);
                Ok(RefCell::into_inner(value))
            }
            Err(rc) => {
                let count = Rc::strong_count(&rc) - 1;
                *place = Some(rc);
                Err(count)
            }
        })
    }

    pub fn iter(&self) -> IndexedCollectionIter<'_, T> {
        self.into_iter()
    }

    pub fn iter_mut(&mut self) -> IndexedCollectionIterMut<'_, T> {
        self.into_iter()
    }
}

impl<T: std::fmt::Debug> std::fmt::Debug for IndexedCollection<T> {
    fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
        f.debug_list().entries(self.iter()).finish()
    }
}

pub struct IndexedCollectionIter<'a, T> {
    inner: std::slice::Iter<'a, Option<Rc<RefCell<T>>>>,
}

impl<'a, T> Iterator for IndexedCollectionIter<'a, T> {
    type Item = Ref<'a, T>;

    fn next(&mut self) -> Option<Self::Item> {
        Some(self.inner.next()?.as_deref()?.borrow())
    }
}

impl<'a, T> IntoIterator for &'a IndexedCollection<T> {
    type Item = Ref<'a, T>;
    type IntoIter = IndexedCollectionIter<'a, T>;

    fn into_iter(self) -> Self::IntoIter {
        IndexedCollectionIter {
            inner: self.items.iter(),
        }
    }
}

pub struct IndexedCollectionIterMut<'a, T> {
    inner: std::slice::IterMut<'a, Option<Rc<RefCell<T>>>>,
}

impl<'a, T> Iterator for IndexedCollectionIterMut<'a, T> {
    type Item = RefMut<'a, T>;

    fn next(&mut self) -> Option<Self::Item> {
        Some(self.inner.next()?.as_deref()?.borrow_mut())
    }
}

impl<'a, T> IntoIterator for &'a mut IndexedCollection<T> {
    type Item = RefMut<'a, T>;
    type IntoIter = IndexedCollectionIterMut<'a, T>;

    fn into_iter(self) -> Self::IntoIter {
        IndexedCollectionIterMut {
            inner: self.items.iter_mut(),
        }
    }
}

#[derive(Debug)]
pub enum MaybeImported<T> {
    Imported(ImportPath),
    Local(T),
}

#[derive(Debug)]
pub struct Element {
    pub offset: Expression,
    pub init: Vec<Rc<RefCell<Function>>>,
}

#[derive(CustomDebug)]
pub struct Data {
    pub offset: Expression,
    #[debug(with = "custom_debug::hexbuf_str")]
    pub init: Blob<Vec<u8>>,
}

#[derive(Debug)]
pub struct Function {
    pub ty: Rc<RefCell<FuncType>>,
    pub body: MaybeImported<Blob<Func>>,
    pub export_name: Option<String>,
}

#[derive(Debug)]
pub struct Global {
    pub ty: GlobalType,
    pub init: MaybeImported<Expression>,
    pub export_name: Option<String>,
}

#[derive(Debug)]
pub struct Memory {
    pub ty: MemType,
    pub import_path: Option<ImportPath>,
    pub export_name: Option<String>,
    pub init: Vec<Data>,
}

#[derive(Debug)]
pub struct Table {
    pub ty: TableType,
    pub import_path: Option<ImportPath>,
    pub export_name: Option<String>,
    pub init: Vec<Element>,
}

#[derive(Debug, Default)]
pub struct Module {
    pub types: IndexedCollection<FuncType>,
    pub functions: IndexedCollection<Function>,
    pub tables: IndexedCollection<Table>,
    pub memories: IndexedCollection<Memory>,
    pub globals: IndexedCollection<Global>,
    pub start: Option<Rc<RefCell<Function>>>,
    pub custom: Vec<Blob<CustomSection>>,
}

impl From<super::module::Module> for Module {
    fn from(src: super::module::Module) -> Self {
        let mut dest = Self::default();
        for section in src.sections {
            match section {
                Section::Custom(custom) => {
                    dest.custom.push(custom);
                }
                Section::Type(Blob(types)) => {
                    dest.types = IndexedCollection::from_iter(types);
                }
                Section::Import(Blob(imports)) => {
                    for import in imports {
                        macro_rules! import {
                            ($dest:ident, $expr:expr) => {
                                dest.$dest.items.push(to_item($expr))
                            };
                        }
                        match import.desc {
                            ImportDesc::Func(type_idx) => import!(
                                functions,
                                Function {
                                    ty: Rc::clone(
                                        dest.types.get_rc_refcell(type_idx.index).unwrap()
                                    ),
                                    body: MaybeImported::Imported(import.path),
                                    export_name: None,
                                }
                            ),
                            ImportDesc::Table(ty) => import!(
                                tables,
                                Table {
                                    ty,
                                    import_path: Some(import.path),
                                    init: Vec::new(),
                                    export_name: None,
                                }
                            ),
                            ImportDesc::Mem(ty) => import!(
                                memories,
                                Memory {
                                    ty,
                                    import_path: Some(import.path),
                                    init: Vec::new(),
                                    export_name: None,
                                }
                            ),
                            ImportDesc::Global(ty) => import!(
                                globals,
                                Global {
                                    ty,
                                    init: MaybeImported::Imported(import.path),
                                    export_name: None,
                                }
                            ),
                        }
                    }
                }
                Section::Function(Blob(type_indices)) => {
                    let types = &dest.types;
                    dest.functions
                        .items
                        .extend(type_indices.into_iter().map(|type_idx| {
                            to_item(Function {
                                ty: Rc::clone(types.get_rc_refcell(type_idx.index).unwrap()),
                                body: MaybeImported::Local(Default::default()),
                                export_name: None,
                            })
                        }));
                }
                Section::Table(Blob(table_types)) => {
                    dest.tables.items.extend(table_types.into_iter().map(|ty| {
                        to_item(Table {
                            ty,
                            import_path: None,
                            export_name: None,
                            init: Vec::new(),
                        })
                    }));
                }
                Section::Memory(Blob(mem_types)) => {
                    dest.memories.items.extend(mem_types.into_iter().map(|ty| {
                        to_item(Memory {
                            ty,
                            import_path: None,
                            export_name: None,
                            init: Vec::new(),
                        })
                    }));
                }
                Section::Global(Blob(globals)) => {
                    dest.globals.items.extend(globals.into_iter().map(|global| {
                        to_item(Global {
                            ty: global.ty,
                            init: MaybeImported::Local(global.init),
                            export_name: None,
                        })
                    }));
                }
                Section::Export(Blob(exports)) => {
                    for export in exports {
                        macro_rules! export {
                            ($dest:ident, $idx:expr) => {
                                dest.$dest.get_mut($idx.index).unwrap().export_name =
                                    Some(export.name)
                            };
                        }
                        match export.desc {
                            ExportDesc::Func(idx) => export!(functions, idx),
                            ExportDesc::Table(idx) => export!(tables, idx),
                            ExportDesc::Mem(idx) => export!(memories, idx),
                            ExportDesc::Global(idx) => export!(globals, idx),
                        }
                    }
                }
                Section::Start(Blob(func_idx)) => {
                    dest.start = Some(Rc::clone(
                        dest.functions.get_rc_refcell(func_idx.index).unwrap(),
                    ));
                }
                Section::Element(Blob(elements)) => {
                    let functions = &dest.functions;
                    for elem in elements {
                        dest.tables
                            .get_mut(elem.table.index)
                            .unwrap()
                            .init
                            .push(Element {
                                offset: elem.offset,
                                init: elem
                                    .init
                                    .into_iter()
                                    .map(|func_idx| {
                                        Rc::clone(functions.get_rc_refcell(func_idx.index).unwrap())
                                    })
                                    .collect(),
                            })
                    }
                }
                Section::Code(Blob(code)) => {
                    for (code, mut func) in code.into_iter().zip(&mut dest.functions) {
                        func.body = MaybeImported::Local(code);
                    }
                }
                Section::Data(Blob(data)) => {
                    for data in data {
                        dest.memories
                            .get_mut(data.memory.index)
                            .unwrap()
                            .init
                            .push(Data {
                                offset: data.offset,
                                init: data.init,
                            })
                    }
                }
            }
        }
        dest
    }
}
<commit_msg>Require mutable collection access in get_mut<commit_after>use crate::builtins::Blob;
use crate::instructions::Expression;
use crate::sections::CustomSection;
use crate::sections::Func;
use crate::sections::{ExportDesc, ImportDesc, ImportPath, Section};
use crate::types::FuncType;
use crate::types::GlobalType;
use crate::types::MemType;
use crate::types::TableType;
use custom_debug::CustomDebug;
use std::cell::{Ref, RefCell, RefMut};
use std::convert::TryFrom;
use std::iter::{Extend, FromIterator};
use std::rc::Rc;

fn to_item<T>(value: T) -> Option<Rc<RefCell<T>>> {
    Some(Rc::new(RefCell::new(value)))
}

pub struct IndexedCollection<T> {
    items: Vec<Option<Rc<RefCell<T>>>>,
    deleted: Vec<u32>,
}

impl<T> Default for IndexedCollection<T> {
    fn default() -> Self {
        Self {
            items: Vec::new(),
            deleted: Vec::new(),
        }
    }
}

impl<T> FromIterator<T> for IndexedCollection<T> {
    fn from_iter<I: IntoIterator<Item = T>>(iter: I) -> Self {
        IndexedCollection {
            items: iter.into_iter().map(to_item).collect(),
            deleted: Vec::new(),
        }
    }
}

impl<T> IndexedCollection<T> {
    fn get_rc_refcell(&self, index: u32) -> Option<&Rc<RefCell<T>>> {
        self.items.get(usize::try_from(index).unwrap())?.as_ref()
    }

    fn get_refcell(&self, index: u32) -> Option<&RefCell<T>> {
        self.get_rc_refcell(index).map(|r| &**r)
    }

    pub fn get(&self, index: u32) -> Option<Ref<T>> {
        self.get_refcell(index).map(RefCell::borrow)
    }

    pub fn get_mut(&mut self, index: u32) -> Option<RefMut<T>> {
        self.get_refcell(index).map(RefCell::borrow_mut)
    }

    pub fn push(&mut self, value: T) -> u32 {
        match self.deleted.pop() {
            Some(index) => {
                self.items[usize::try_from(index).unwrap()] = to_item(value);
                index
            }
            None => {
                let index = u32::try_from(self.items.len()).unwrap();
                self.items.push(to_item(value));
                index
            }
        }
    }

    // Don't allow removals for referenced elements.
    // Will return a number of references as error (excluding the owner).
    pub fn try_remove(&mut self, index: u32) -> Option<Result<T, usize>> {
        let place = self.items.get_mut(usize::try_from(index).unwrap())?;
        Some(match Rc::try_unwrap(place.take()?) {
            Ok(value) => {
                self.deleted.push(index);
                Ok(RefCell::into_inner(value))
            }
            Err(rc) => {
                let count = Rc::strong_count(&rc) - 1;
                *place = Some(rc);
                Err(count)
            }
        })
    }

    pub fn iter(&self) -> IndexedCollectionIter<'_, T> {
        self.into_iter()
    }

    pub fn iter_mut(&mut self) -> IndexedCollectionIterMut<'_, T> {
        self.into_iter()
    }
}

impl<T: std::fmt::Debug> std::fmt::Debug for IndexedCollection<T> {
    fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
        f.debug_list().entries(self.iter()).finish()
    }
}

pub struct IndexedCollectionIter<'a, T> {
    inner: std::slice::Iter<'a, Option<Rc<RefCell<T>>>>,
}

impl<'a, T> Iterator for IndexedCollectionIter<'a, T> {
    type Item = Ref<'a, T>;

    fn next(&mut self) -> Option<Self::Item> {
        Some(self.inner.next()?.as_deref()?.borrow())
    }
}

impl<'a, T> IntoIterator for &'a IndexedCollection<T> {
    type Item = Ref<'a, T>;
    type IntoIter = IndexedCollectionIter<'a, T>;

    fn into_iter(self) -> Self::IntoIter {
        IndexedCollectionIter {
            inner: self.items.iter(),
        }
    }
}

pub struct IndexedCollectionIterMut<'a, T> {
    inner: std::slice::IterMut<'a, Option<Rc<RefCell<T>>>>,
}

impl<'a, T> Iterator for IndexedCollectionIterMut<'a, T> {
    type Item = RefMut<'a, T>;

    fn next(&mut self) -> Option<Self::Item> {
        Some(self.inner.next()?.as_deref()?.borrow_mut())
    }
}

impl<'a, T> IntoIterator for &'a mut IndexedCollection<T> {
    type Item = RefMut<'a, T>;
    type IntoIter = IndexedCollectionIterMut<'a, T>;

    fn into_iter(self) -> Self::IntoIter {
        IndexedCollectionIterMut {
            inner: self.items.iter_mut(),
        }
    }
}

#[derive(Debug)]
pub enum MaybeImported<T> {
    Imported(ImportPath),
    Local(T),
}

#[derive(Debug)]
pub struct Element {
    pub offset: Expression,
    pub init: Vec<Rc<RefCell<Function>>>,
}

#[derive(CustomDebug)]
pub struct Data {
    pub offset: Expression,
    #[debug(with = "custom_debug::hexbuf_str")]
    pub init: Blob<Vec<u8>>,
}

#[derive(Debug)]
pub struct Function {
    pub ty: Rc<RefCell<FuncType>>,
    pub body: MaybeImported<Blob<Func>>,
    pub export_name: Option<String>,
}

#[derive(Debug)]
pub struct Global {
    pub ty: GlobalType,
    pub init: MaybeImported<Expression>,
    pub export_name: Option<String>,
}

#[derive(Debug)]
pub struct Memory {
    pub ty: MemType,
    pub import_path: Option<ImportPath>,
    pub export_name: Option<String>,
    pub init: Vec<Data>,
}

#[derive(Debug)]
pub struct Table {
    pub ty: TableType,
    pub import_path: Option<ImportPath>,
    pub export_name: Option<String>,
    pub init: Vec<Element>,
}

#[derive(Debug, Default)]
pub struct Module {
    pub types: IndexedCollection<FuncType>,
    pub functions: IndexedCollection<Function>,
    pub tables: IndexedCollection<Table>,
    pub memories: IndexedCollection<Memory>,
    pub globals: IndexedCollection<Global>,
    pub start: Option<Rc<RefCell<Function>>>,
    pub custom: Vec<Blob<CustomSection>>,
}

impl From<super::module::Module> for Module {
    fn from(src: super::module::Module) -> Self {
        let mut dest = Self::default();
        for section in src.sections {
            match section {
                Section::Custom(custom) => {
                    dest.custom.push(custom);
                }
                Section::Type(Blob(types)) => {
                    dest.types = IndexedCollection::from_iter(types);
                }
                Section::Import(Blob(imports)) => {
                    for import in imports {
                        macro_rules! import {
                            ($dest:ident, $expr:expr) => {
                                dest.$dest.items.push(to_item($expr))
                            };
                        }
                        match import.desc {
                            ImportDesc::Func(type_idx) => import!(
                                functions,
                                Function {
                                    ty: Rc::clone(
                                        dest.types.get_rc_refcell(type_idx.index).unwrap()
                                    ),
                                    body: MaybeImported::Imported(import.path),
                                    export_name: None,
                                }
                            ),
                            ImportDesc::Table(ty) => import!(
                                tables,
                                Table {
                                    ty,
                                    import_path: Some(import.path),
                                    init: Vec::new(),
                                    export_name: None,
                                }
                            ),
                            ImportDesc::Mem(ty) => import!(
                                memories,
                                Memory {
                                    ty,
                                    import_path: Some(import.path),
                                    init: Vec::new(),
                                    export_name: None,
                                }
                            ),
                            ImportDesc::Global(ty) => import!(
                                globals,
                                Global {
                                    ty,
                                    init: MaybeImported::Imported(import.path),
                                    export_name: None,
                                }
                            ),
                        }
                    }
                }
                Section::Function(Blob(type_indices)) => {
                    let types = &dest.types;
                    dest.functions
                        .items
                        .extend(type_indices.into_iter().map(|type_idx| {
                            to_item(Function {
                                ty: Rc::clone(types.get_rc_refcell(type_idx.index).unwrap()),
                                body: MaybeImported::Local(Default::default()),
                                export_name: None,
                            })
                        }));
                }
                Section::Table(Blob(table_types)) => {
                    dest.tables.items.extend(table_types.into_iter().map(|ty| {
                        to_item(Table {
                            ty,
                            import_path: None,
                            export_name: None,
                            init: Vec::new(),
                        })
                    }));
                }
                Section::Memory(Blob(mem_types)) => {
                    dest.memories.items.extend(mem_types.into_iter().map(|ty| {
                        to_item(Memory {
                            ty,
                            import_path: None,
                            export_name: None,
                            init: Vec::new(),
                        })
                    }));
                }
                Section::Global(Blob(globals)) => {
                    dest.globals.items.extend(globals.into_iter().map(|global| {
                        to_item(Global {
                            ty: global.ty,
                            init: MaybeImported::Local(global.init),
                            export_name: None,
                        })
                    }));
                }
                Section::Export(Blob(exports)) => {
                    for export in exports {
                        macro_rules! export {
                            ($dest:ident, $idx:expr) => {
                                dest.$dest.get_mut($idx.index).unwrap().export_name =
                                    Some(export.name)
                            };
                        }
                        match export.desc {
                            ExportDesc::Func(idx) => export!(functions, idx),
                            ExportDesc::Table(idx) => export!(tables, idx),
                            ExportDesc::Mem(idx) => export!(memories, idx),
                            ExportDesc::Global(idx) => export!(globals, idx),
                        }
                    }
                }
                Section::Start(Blob(func_idx)) => {
                    dest.start = Some(Rc::clone(
                        dest.functions.get_rc_refcell(func_idx.index).unwrap(),
                    ));
                }
                Section::Element(Blob(elements)) => {
                    let functions = &dest.functions;
                    for elem in elements {
                        dest.tables
                            .get_mut(elem.table.index)
                            .unwrap()
                            .init
                            .push(Element {
                                offset: elem.offset,
                                init: elem
                                    .init
                                    .into_iter()
                                    .map(|func_idx| {
                                        Rc::clone(functions.get_rc_refcell(func_idx.index).unwrap())
                                    })
                                    .collect(),
                            })
                    }
                }
                Section::Code(Blob(code)) => {
                    for (code, mut func) in code.into_iter().zip(&mut dest.functions) {
                        func.body = MaybeImported::Local(code);
                    }
                }
                Section::Data(Blob(data)) => {
                    for data in data {
                        dest.memories
                            .get_mut(data.memory.index)
                            .unwrap()
                            .init
                            .push(Data {
                                offset: data.offset,
                                init: data.init,
                            })
                    }
                }
            }
        }
        dest
    }
}
<|endoftext|>


<commit_before>#[link(name="env", vers="1.0.0", author="LeoTestard")];

/*
 * This file is part of the uutils coreutils package.
 *
 * (c) Jordi Boggiano <j.boggiano@seld.be>
 *
 * For the full copyright and license information, please view the LICENSE
 * file that was distributed with this source code.
 */

/* last synced with: env (GNU coreutils) 8.13 */

struct options {
    ignore_env: bool,
    null: bool,
    unsets: ~[~str],
    sets: ~[(~str, ~str)],
    program: ~[~str]
}

fn usage(prog: &str) {
    println!("Usage: {:s} [OPTION]... [-] [NAME=VALUE]... [COMMAND [ARG]...]", prog);
    println!("Set each NAME to VALUE in the environment and run COMMAND\n");
    println!("Possible options are:");
    println!("  -i --ignore-environment\t start with an empty environment");
    println!("  -0 --null              \t end each output line with a 0 byte rather than newline");
    println!("  -u --unset NAME        \t remove variable from the environment");
    println!("  -h --help              \t display this help and exit");
    println!("  -V --version           \t output version information and exit\n");
    println!("A mere - implies -i. If no COMMAND, print the resulting environment");
}

fn version() {
    println!("env 1.0.0");
}

// print name=value env pairs on screen
// if null is true, separate pairs with a \0, \n otherwise
fn print_env(null: bool) {
    let env = std::os::env();

    for &(ref n, ref v) in env.iter() {
        print!("{:s}={:s}{:c}",
            n.as_slice(),
            v.as_slice(),
            if null { '\0' } else { '\n' }
        );
    }
}

fn main() {
    let args = std::os::args();
    let prog = args[0].as_slice();

    // to handle arguments the same way than GNU env, we can't use getopts
    let mut opts = ~options {
        ignore_env: false,
        null: false,
        unsets: ~[],
        sets: ~[],
        program: ~[]
    };

    let mut wait_cmd = false;
    let mut iter = args.iter();
    iter.next(); // skip program
    let mut item = iter.next();

    // the for loop doesn't work here,
    // because we need sometines to read 2 items forward,
    // and the iter can't be borrowed twice
    while item != None {
        let opt = item.unwrap();

        if wait_cmd {
            // we still accept NAME=VAL here but not other options
            let mut sp = opt.splitn_iter('=', 1);
            let name = sp.next();
            let value = sp.next();

            match (name, value) {
                (Some(n), Some(v)) => {
                    opts.sets.push((n.into_owned(), v.into_owned()));
                }
                _ => {
                    // read the program now
                    opts.program.push(opt.to_owned());
                    break;
                }
            }
        } else if opt.starts_with("--") {
            match *opt {
                ~"--help" => { usage(prog); return }
                ~"--version" => { version(); return }

                ~"--ignore-environment" => opts.ignore_env = true,
                ~"--null" => opts.null = true,
                ~"--unset" => {
                    let var = iter.next();

                    match var {
                        None => println!("{:s}: this option requires an argument: {:s}", prog, opt.as_slice()),
                        Some(s) => opts.unsets.push(s.to_owned())
                    }
                }

                _ => {
                    println!("{:s}: invalid option \"{:s}\"", prog, *opt);
                    println!("Type \"{:s} --help\" for detailed informations", prog);
                    return
                }
            }
        } else if opt.starts_with("-") {
            if opt.len() == 0 {
                // implies -i and stop parsing opts
                wait_cmd = true;
                opts.ignore_env = true;
                continue;
            }

            let mut chars = opt.iter();
            chars.next();

            for c in chars {
                // short versions of options
                match c {
                    'h' => { usage(prog); return }
                    'V' => { version(); return }
                    'i' => opts.ignore_env = true,
                    '0' => opts.null = true,
                    'u' => {
                        let var = iter.next();

                        match var {
                            None => println!("{:s}: this option requires an argument: {:s}", prog, opt.as_slice()),
                            Some(s) => opts.unsets.push(s.to_owned())
                        }
                    }
                    _ => {
                        println!("{:s}: illegal option -- {:c}", prog, c);
                        println!("Type \"{:s} --help\" for detailed informations", prog);
                        return
                    }
                }
            }
        } else {
            // is it a NAME=VALUE like opt ?
            let mut sp = opt.splitn_iter('=', 1);
            let name = sp.next();
            let value = sp.next();

            match (name, value) {
                (Some(n), Some(v)) => {
                    // yes
                    opts.sets.push((n.into_owned(), v.into_owned()));
                    wait_cmd = true;
                }
                // no, its a program-like opt
                _ => {
                    opts.program.push(opt.to_owned());
                    break;
                }
            }
        }

        item = iter.next();
    }

    // read program arguments
    for opt in iter {
        opts.program.push(opt.to_owned());
    }

    let env = std::os::env();

    if opts.ignore_env {
        for &(ref name, _) in env.iter() {
            std::os::unsetenv(name.as_slice())
        }
    }

    for ref name in opts.unsets.iter() {
        std::os::unsetenv(name.as_slice())
    }

    for &(ref name, ref val) in opts.sets.iter() {
        std::os::setenv(name.as_slice(), val.as_slice())
    }

    match opts.program {
        [ref prog, ..args] => {
            let status = std::run::process_status(prog.as_slice(), args.as_slice());
            std::os::set_exit_status(status)
        }

        [] => {
            // no program provided
            print_env(opts.null);
        }
    }
}
<commit_msg>Use new ProcessExit enum<commit_after>#[link(name="env", vers="1.0.0", author="LeoTestard")];

/*
 * This file is part of the uutils coreutils package.
 *
 * (c) Jordi Boggiano <j.boggiano@seld.be>
 *
 * For the full copyright and license information, please view the LICENSE
 * file that was distributed with this source code.
 */

/* last synced with: env (GNU coreutils) 8.13 */

struct options {
    ignore_env: bool,
    null: bool,
    unsets: ~[~str],
    sets: ~[(~str, ~str)],
    program: ~[~str]
}

fn usage(prog: &str) {
    println!("Usage: {:s} [OPTION]... [-] [NAME=VALUE]... [COMMAND [ARG]...]", prog);
    println!("Set each NAME to VALUE in the environment and run COMMAND\n");
    println!("Possible options are:");
    println!("  -i --ignore-environment\t start with an empty environment");
    println!("  -0 --null              \t end each output line with a 0 byte rather than newline");
    println!("  -u --unset NAME        \t remove variable from the environment");
    println!("  -h --help              \t display this help and exit");
    println!("  -V --version           \t output version information and exit\n");
    println!("A mere - implies -i. If no COMMAND, print the resulting environment");
}

fn version() {
    println!("env 1.0.0");
}

// print name=value env pairs on screen
// if null is true, separate pairs with a \0, \n otherwise
fn print_env(null: bool) {
    let env = std::os::env();

    for &(ref n, ref v) in env.iter() {
        print!("{:s}={:s}{:c}",
            n.as_slice(),
            v.as_slice(),
            if null { '\0' } else { '\n' }
        );
    }
}

fn main() {
    let args = std::os::args();
    let prog = args[0].as_slice();

    // to handle arguments the same way than GNU env, we can't use getopts
    let mut opts = ~options {
        ignore_env: false,
        null: false,
        unsets: ~[],
        sets: ~[],
        program: ~[]
    };

    let mut wait_cmd = false;
    let mut iter = args.iter();
    iter.next(); // skip program
    let mut item = iter.next();

    // the for loop doesn't work here,
    // because we need sometines to read 2 items forward,
    // and the iter can't be borrowed twice
    while item != None {
        let opt = item.unwrap();

        if wait_cmd {
            // we still accept NAME=VAL here but not other options
            let mut sp = opt.splitn_iter('=', 1);
            let name = sp.next();
            let value = sp.next();

            match (name, value) {
                (Some(n), Some(v)) => {
                    opts.sets.push((n.into_owned(), v.into_owned()));
                }
                _ => {
                    // read the program now
                    opts.program.push(opt.to_owned());
                    break;
                }
            }
        } else if opt.starts_with("--") {
            match *opt {
                ~"--help" => { usage(prog); return }
                ~"--version" => { version(); return }

                ~"--ignore-environment" => opts.ignore_env = true,
                ~"--null" => opts.null = true,
                ~"--unset" => {
                    let var = iter.next();

                    match var {
                        None => println!("{:s}: this option requires an argument: {:s}", prog, opt.as_slice()),
                        Some(s) => opts.unsets.push(s.to_owned())
                    }
                }

                _ => {
                    println!("{:s}: invalid option \"{:s}\"", prog, *opt);
                    println!("Type \"{:s} --help\" for detailed informations", prog);
                    return
                }
            }
        } else if opt.starts_with("-") {
            if opt.len() == 0 {
                // implies -i and stop parsing opts
                wait_cmd = true;
                opts.ignore_env = true;
                continue;
            }

            let mut chars = opt.iter();
            chars.next();

            for c in chars {
                // short versions of options
                match c {
                    'h' => { usage(prog); return }
                    'V' => { version(); return }
                    'i' => opts.ignore_env = true,
                    '0' => opts.null = true,
                    'u' => {
                        let var = iter.next();

                        match var {
                            None => println!("{:s}: this option requires an argument: {:s}", prog, opt.as_slice()),
                            Some(s) => opts.unsets.push(s.to_owned())
                        }
                    }
                    _ => {
                        println!("{:s}: illegal option -- {:c}", prog, c);
                        println!("Type \"{:s} --help\" for detailed informations", prog);
                        return
                    }
                }
            }
        } else {
            // is it a NAME=VALUE like opt ?
            let mut sp = opt.splitn_iter('=', 1);
            let name = sp.next();
            let value = sp.next();

            match (name, value) {
                (Some(n), Some(v)) => {
                    // yes
                    opts.sets.push((n.into_owned(), v.into_owned()));
                    wait_cmd = true;
                }
                // no, its a program-like opt
                _ => {
                    opts.program.push(opt.to_owned());
                    break;
                }
            }
        }

        item = iter.next();
    }

    // read program arguments
    for opt in iter {
        opts.program.push(opt.to_owned());
    }

    let env = std::os::env();

    if opts.ignore_env {
        for &(ref name, _) in env.iter() {
            std::os::unsetenv(name.as_slice())
        }
    }

    for ref name in opts.unsets.iter() {
        std::os::unsetenv(name.as_slice())
    }

    for &(ref name, ref val) in opts.sets.iter() {
        std::os::setenv(name.as_slice(), val.as_slice())
    }

    match opts.program {
        [ref prog, ..args] => {
            let exit = std::run::process_status(prog.as_slice(), args.as_slice());
            match exit {
                std::rt::io::process::ExitStatus(status) => std::os::set_exit_status(status),
                _ => std::os::set_exit_status(1)
            }
        }

        [] => {
            // no program provided
            print_env(opts.null);
        }
    }
}
<|endoftext|>