### A Pluto.jl notebook ###
# v0.15.1

using Markdown
using InteractiveUtils

# ╔═╡ 8b92a6cf-cd6a-4364-9f59-598bcf206ef1
html"<button onclick=present()>Presentation mode</button>"

# ╔═╡ 5c402564-e0e1-11eb-105b-951cfa19cacd
md"""
# Julia for Statistics and Data Science

- *Cecile Ane*, *Douglas Bates* & *Claudia Solis-Lemus* all with U. of Wisconsin - Madison

1. What is **Julia**?
    - How does it differ from other languages, particularly **R** and **Python**, used for data science?
    - How do I get started?
2. A deep dive into a shallow function - how does compilation work?
3. Motivating example -- `MixedModels.jl` compared to R's `lme4`
4. Some recommended packages for Statistics and Data Science
5. `RCall.jl` and `PyCall.jl`: to allow you to build on your current skills
"""

# ╔═╡ 7e8e3903-a76d-4310-b5df-8508fdd7dba3
md"""
## What is **Julia**?

- an Open Source language designed for "technical computing".  The motivation is best explained in [this blog post](https://julialang.org/blog/2012/02/why-we-created-julia/) by its creators.
- development of the [language itself](https://github.com/JuliaLang/julia) and most packages, e.g. [DataFrames](https://github.com/JuliaData/DataFrames.jl), takes place on github.
- a few things to notice about [github.com/JuliaLang/julia](https://github.com/JuliaLang/julia)
    - nearly 1100 contributors
    - over 40,000 issues or pull requests
    - over 50,000 commits
    - the most common coding language used in the `julia` repository is Julia
- installation is easiest using binary downloads from [julialang.org](https://julialang.org/downloads)
"""

# ╔═╡ dc65b3ed-2acf-4bf5-be2e-77dde963a7f1
md"""
## Package Ecosystem
- Julia packages are public git repositories (usually on `github.com` or `gitlab.com`) listed in the [General Registry](https://github.com/JuliaRegistries/General)
- see the [Packages tab at julialang.org](https://julialang.org/packages) for links to package exploration tools.  I prefer the one at [JuliaHub](https://juliahub.com/ui/packages)
    + when browsing Julia packages visit the repository and look at the Languages bar on the right hand side.  Most packages are 100% Julia code.  As [Erik Engheim wrote](https://medium.com/codex/is-julia-really-fast-12cd7caef96b), "it's turtles all the way down".
    + [this blog posting](https://julialang.org/blog/2021/08/general-survey/) provides some analysis of the package registry contents
- setting version numbers according to [semantic versioning](https://semver.org/) is strongly encouraged
- often packages are maintained by organizations on github.com, e.g. DataFrames.jl belongs to the JuliaData organization.
"""

# ╔═╡ bb892d27-63f6-4267-bfcc-758e743c31d2
md"""
## Ways to use Julia
1. Start the App and use the Read-Eval-Print-Loop (REPL) **[demo this]**
    - the REPL has name completion and other conveniences
    - the REPL has "modes"
        + standard Julia input
        + help mode
        + package manager mode
        + shell mode
        + some packages, e.g. `RCall`, enable their own modes
2. An Integrated Development Environment (IDE)
    - [Visual Studio Code](https://code.visualstudio.com/) (with the Julia extension) is the most widely used
3. [Jupyter notebooks](https://jupyter.org) using the [IJulia package](https://github.com/JuliaLang/IJulia.jl)
4. [Pluto](https://github.com/fonsp/Pluto.jl) notebooks, which we are using here.
    - (+) notebooks are self-contained with regard to packages needed and versions thereof
    - (+) notebooks are reactive - changes in one place are propagated to others
    - (-) reactivity sometimes requires awkward code phrasing
    - (-) output displayed is the result of `show` on the value of a block, which sometimes misses things that would normally be displayed
"""

# ╔═╡ 864522f6-a214-4397-91cb-8d4d670e7638
md"""
## Why Julia?
- allows you to bypass the **two language problem**
    + high-level, *dynamic* languages (R, Python, Matlab/Octave, etc.) allow for ease of use and high productivity
    + often code in these languages is unable to give the performance required for high-throughput, production uses
    + the usual solution is to re-write the performance-critical parts in a low-level, *static* language (C, C++, Fortran) and integrate that with the higher-level code
- language and support system are carefully crafted to provide **both** high level tools and high performance
- recently developed language with the benefit of hindsight and modern tools.
"""

# ╔═╡ d78b2581-31cd-4a08-9c8f-59aedc5c3086
md"""
## Why not Julia?
- recently developed language
    + ecosystem is not as mature as R, Python, etc.
    + don't have time to learn a new language and support system because this proposal is due Friday
"""

# ╔═╡ 0aab07a1-893e-4c8b-ae8d-198146058d2e
md"""
## In what ways is **Julia** similar to **R**?

- Algorithms are expressed as **functions**
- Functions are *generic* - different *methods* can be defined for argument *signatures*
    * in **R** a function must be explicitly designated as a generic function
    * **R** has different systems of generic functions and methods (S3 and S4)
        - S3 method dispatch is on the first argument's class only (single dispatch)
        - S4 allows for **multiple dispatch**
- In Julia all functions are generic and all functions use multiple dispatch
"""

# ╔═╡ 85fcacb1-7ad2-4fb8-9c87-28cfaa0f1404
md"""
## What makes Julia fast?
- The first time a particular method is invoked, it is compiled using a *Just In Time* compiler based on [**LLVM**](https://llvm.org) - the low-level virtual machine
    * good news - the actual execution is fast and subsequent calls are fast
    * bad news - the first call can be slow because of the compilation overhead
        + known informally as the "Time To First Plot" problem
- Ease of programming and ease of compilation are often antithetical goals
    * ease of programming emphasizes generality provided by languages with dynamic types
    * ease of compilation requires very specific, usually static, types b/c that's what the processor works with
- In Julia methods are often defined for general, abstract argument types but compilation takes place for specific, concrete types.

Time for a deep dive (notebook named `2compilation.jl`)
"""

# ╔═╡ Cell order:
# ╟─8b92a6cf-cd6a-4364-9f59-598bcf206ef1
# ╟─5c402564-e0e1-11eb-105b-951cfa19cacd
# ╟─7e8e3903-a76d-4310-b5df-8508fdd7dba3
# ╟─dc65b3ed-2acf-4bf5-be2e-77dde963a7f1
# ╟─bb892d27-63f6-4267-bfcc-758e743c31d2
# ╟─864522f6-a214-4397-91cb-8d4d670e7638
# ╟─d78b2581-31cd-4a08-9c8f-59aedc5c3086
# ╟─0aab07a1-893e-4c8b-ae8d-198146058d2e
# ╟─85fcacb1-7ad2-4fb8-9c87-28cfaa0f1404


"""
    sign_cadlag(x::N)::N where {N<:Real}

This function works like the sign function but is ``1`` for input ``0``.

### Input

- `x` -- real scalar

### Output

``1`` if ``x ≥ 0``, ``-1`` otherwise.

### Notes

This is the sign function right-continuous at zero (see
[càdlàg function](https://en.wikipedia.org/wiki/C%C3%A0dl%C3%A0g)).
It can be used with vector-valued arguments via the dot operator.

### Examples

```jldoctest
julia> LazySets.sign_cadlag.([-0.6, 1.3, 0.0])
3-element Array{Float64,1}:
 -1.0
  1.0
  1.0
```
"""
function sign_cadlag(x::N)::N where {N<:Real}
    return x < zero(x) ? -one(x) : one(x)
end

"""
    substitute(substitution::Dict{Int, T}, x::AbstractVector{T}) where {T}

Apply a substitution to a given vector.

### Input

- `substitution` -- substitution (a mapping from an index to a new value)
- `x`            -- vector

### Output

A fresh vector corresponding to `x` after `substitution` was applied.
"""
function substitute(substitution::Dict{Int, T}, x::AbstractVector{T}) where {T}
    return substitute!(substitution, copy(x))
end

"""
    substitute!(substitution::Dict{Int, T}, x::AbstractVector{T}) where {T}

Apply a substitution to a given vector.

### Input

- `substitution` -- substitution (a mapping from an index to a new value)
- `x`            -- vector (modified in this function)

### Output

The same (but see the Notes below) vector `x` but after `substitution` was
applied.

### Notes

The vector `x` is modified in-place if it has type `Vector` or `SparseVector`.
Otherwise, we first create a new `Vector` from it.
"""
function substitute!(substitution::Dict{Int, T}, x::AbstractVector{T}) where {T}
    return substitute!(Vector(x), substitution)
end

function substitute!(substitution::Dict{Int, T},
                     x::Union{Vector{T}, SparseVector{T}}) where {T}
    for (index, value) in substitution
        x[index] = value
    end
    return x
end

"""
    reseed(rng::AbstractRNG, seed::Union{Int, Nothing})::AbstractRNG

Reset the RNG seed if the seed argument is a number.

### Input

- `rng`  -- random number generator
- `seed` -- seed for reseeding

### Output

The input RNG if the seed is `nothing`, and a reseeded RNG otherwise.
"""
function reseed(rng::AbstractRNG, seed::Union{Int, Nothing})::AbstractRNG
    if seed != nothing
        return Random.seed!(rng, seed)
    end
    return rng
end

"""
    StrictlyIncreasingIndices

Iterator over the vectors of `m` strictly increasing indices from 1 to `n`.

### Fields

- `n` -- size of the index domain
- `m` -- number of indices to choose (resp. length of the vectors)

### Notes

The vectors are modified in-place.

The iterator ranges over ``\\binom{n}{m}`` (`n` choose `m`) possible vectors.

This implementation results in a lexicographic order with the last index growing
first.

### Examples

```jldoctest
julia> for v in LazySets.StrictlyIncreasingIndices(4, 2)
           println(v)
       end
[1, 2]
[1, 3]
[1, 4]
[2, 3]
[2, 4]
[3, 4]
```
"""
struct StrictlyIncreasingIndices
    n::Int
    m::Int

    function StrictlyIncreasingIndices(n::Int, m::Int)
        @assert n >= m > 0 "require n >= m > 0"
        new(n, m)
    end
end

Base.eltype(::Type{StrictlyIncreasingIndices}) = Vector{Int}
Base.length(sii::StrictlyIncreasingIndices) = binomial(sii.n, sii.m)

# initialization
function Base.iterate(sii::StrictlyIncreasingIndices)
    v = [1:sii.m;]
    return (v, v)
end

# normal iteration
function Base.iterate(sii::StrictlyIncreasingIndices, state::AbstractVector{Int})
    v = state
    i = sii.m
    diff = sii.n
    if i == diff
        return nothing
    end
    while v[i] == diff
        i -= 1
        diff -= 1
    end
    # update vector
    v[i] += 1
    for j in i+1:sii.m
        v[j] = v[j-1] + 1
    end
    # detect termination: first index has maximum value
    if i == 1 && v[1] == (sii.n - sii.m + 1)
        return (v, nothing)
    end
    return (v, v)
end

# termination
function Base.iterate(sii::StrictlyIncreasingIndices, state::Nothing)
    return nothing
end

"""
    subtypes(interface, concrete::Bool)

Return the concrete subtypes of a given interface.

### Input

- `interface` -- an abstract type, usually a set interface
- `concrete`  -- if `true`, seek further the inner abstract subtypes of the given
                 interface, otherwise return only the direct subtypes of `interface`

### Output

A list with the subtypes of the abstract type `interface`, sorted alphabetically.

### Examples

Consider the `AbstractPolytope` interface. If we include the abstract subtypes
of this interface,

```jldoctest subtypes
julia> using LazySets: subtypes

julia> subtypes(AbstractPolytope, false)
4-element Array{Any,1}:
 AbstractCentrallySymmetricPolytope
 AbstractPolygon
 HPolytope
 VPolytope
```

We can use this function to obtain the concrete subtypes of
`AbstractCentrallySymmetricPolytope` and `AbstractPolygon` (further until all
concrete types are obtained), using the `concrete` flag:

```jldoctest subtypes
julia> subtypes(AbstractPolytope, true)
14-element Array{Type,1}:
 Ball1
 BallInf
 HPolygon
 HPolygonOpt
 HPolytope
 Hyperrectangle
 Interval
 LineSegment
 Singleton
 SymmetricIntervalHull
 VPolygon
 VPolytope
 ZeroSet
 Zonotope
```
"""
function subtypes(interface, concrete::Bool)

    subtypes_to_test = subtypes(interface)

    # do not seek the concrete subtypes further
    if !concrete
        return sort(subtypes_to_test, by=string)
    end

    result = Vector{Type}()
    i = 0
    while i < length(subtypes_to_test)
        i += 1
        subtype = subtypes_to_test[i]
        new_subtypes = subtypes(subtype)
        if isempty(new_subtypes)
            # base type found
            push!(result, subtype)
        else
            # yet another interface layer
            append!(subtypes_to_test, new_subtypes)
        end
    end
    return sort(result, by=string)
end


"""
Read CSV-formatted posterior output.

# API

See [`StanCSV.read_chain`](@ref) and [`StanCSV.matching_files`], neither is exported.

# File format

Posterior samples are supposed to be available in a CSV file which follows the following
conventions:

1. Whitespace and content after `#` is ignored. No fields are quoted or escaped. If a line
has a `#`, it cannot have content.

2. The first non-ignored line contains a comma-separated list of variable names, of the
format `var[.i1.i2.…]` where optional indexes follow dots. Indexes for the same variable are
contiguous, and have a column-major layout.

3. Subsequent lines contain the same number of comma-separated floating-point values.

`cmdstan` outputs posterior samples in this format (diagnostics are ignored for now), hence
the name.
"""
module StanCSV

using ArgCheck: @argcheck
using DocStringExtensions: FUNCTIONNAME, SIGNATURES
import ..Chains

####
#### Header parsing building blocks
####

"""
$(SIGNATURES)

Parse a Stan variable name as `name::Symbol => indexes::Tuple`.
"""
function parse_variable_name(variable_name::AbstractString)
    parts = split(variable_name, '.')
    @argcheck !isempty(first(parts))
    name = Symbol(strip(first(parts)))
    index = tuple(parse.(Ref(Int), parts[2:end])...)
    name => index
end

"""
$(SIGNATURES)

Given a vector of `name => index` pairs and a starting index `i`, which should be `1`s, find
the last index for this variable (ie the dimensions) and check that that intermediate
indices are contiguous (in column major ordering).

Return `name`, `dimensions`, and the position for the continuation.
"""
function collapse_contiguous_dimensions(names_indexes, i)
    name, index = names_indexes[i]
    @argcheck all(index .== 1) "Indexes for $(name) don't start with ones."
    n = length(names_indexes)
    j = i + 1
    while j ≤ n
        if names_indexes[j][1] ≠ name
            break
        else
            j += 1
        end
    end
    dimensions = names_indexes[j - 1][2]
    indexes = CartesianIndices(dimensions)
    for (k, ix) in zip(i:(j-1), CartesianIndices(dimensions))
        index_k = names_indexes[k][2]
        @argcheck CartesianIndex(index_k) == ix "Non-contiguous index $(k) for $(name)."
    end
    name, dimensions, j
end

"""
$(SIGNATURES)

Parse a vector of `names => indexes` to a column schema.
"""
function parse_schema(names_indexes)
    n = length(names_indexes)
    i = 1
    schema = Pair{Symbol,Tuple{Vararg{Int}}}[]
    while i ≤ n
        name, dimensions, next_i = collapse_contiguous_dimensions(names_indexes, i)
        @argcheck all(s -> first(schema) ≠ name, schema) "Duplicate name $(name) in column $(i)."
        push!(schema, name => dimensions)
        i = next_i
    end
    Chains.IndexSchema(NamedTuple{Tuple(first.(schema))}(last.(schema)))
end

####
#### CSV file reading.
####

"""
$(SIGNATURES)

Test if `line` is a comment line.
"""
is_comment_line(line) = occursin(r"^ *#", line)

"""
$(SIGNATURES)

Find the first non-comment line and read it as a `Chains.IndexSchema`. When there is no
such line, throw an `EOFError`.
"""
function read_schema(io::IO)
    while true                  # will throw an EOFError if not fount
        line = readline(io; keep = false)
        if !is_comment_line(line)
            return parse_schema(parse_variable_name.(split(line, ',')))
        end
    end
end

"""
$(SIGNATURES)

Read `nfields` fields from `io`, parse as `Float64`, and push into `buffer`, returning
`true`. Fewer fields than `nfields` results in an error.

If a line starts with a `'#'`, do no parsing, return `false`.
"""
function read_csv_line!(buffer, io::IO, nfields::Integer)
    line = readline(io; keep = false)
    is_comment_line(line) && return false
    pos = 1
    last_pos = lastindex(line)
    for _ in 1:nfields
        @argcheck pos ≤ last_pos "Fewer than $(nfields) fields in line."
        delim_pos = something(findnext(isequal(','), line, pos), last_pos + 1)
        push!(buffer, parse(Float64, SubString(line, pos, delim_pos - 1)))
        pos = delim_pos + 1
    end
    true
end

function read_csv_flat_data!(buffer, io, nfields)
    row_count = 0
    while !eof(io)
        row_count += read_csv_line!(buffer, io, nfields)
    end
    row_count
end

function csv_buffer_to_array(buffer, row_count, dims)
    n = length(dims)
    permutedims(reshape(buffer, dims..., row_count), vcat([1 + n], 1:n))
end

function read_chain(io::IO)
    sch = read_schema(io)
    nfields = length(sch)
    buffer = Vector{Float64}()
    row_count = read_csv_flat_data!(buffer, io, nfields)
    sample_matrix = collect(permutedims(reshape(buffer, nfields, row_count)))
    Chains.Chain(sch, sample_matrix)
end

"""
$(SIGNATURES)

Read data from a CSV file, that uses the output format from Stan.

Return a vector of `name => array` pairs, where `name::Symbol` is the variable name, and the
`array` is the data for that variable.

Consecutive columns for the same variable name with `n`-dimensional indexes are assembled
into an `n+1` dimensional array, with index `var[row, i1, i2, …]` containing the results
for `var.i1.i2.…` in the given `row`.
"""
read_chain(filename::AbstractString) = open(read_chain, filename, "r")

"""
$(SIGNATURES)

Return a vector `id::Int => filename` where `filename` is `prefix_<digits>.csv` and the path
is a file. Prefix should be a path, eg

```julia
$(FUNCTIONNAME)("/tmp/samples_")
```

will return `[1 => "/tmp/samples_1.csv", 2 => "/tmp/samples_2.csv", …]` if the files exist.

Recommended use: obtaining all the filenames in a directory.
"""
function matching_files(prefix::AbstractString)
    dir = dirname(prefix)
    base = basename(prefix)
    pattern = Regex("^" * base * raw"(\d+)\.csv$")
    ids_files = Pair{Int,String}[]
    for filename in readdir(dir)
        m = match(pattern, filename)
        p = joinpath(dir, filename)
        if m ≢ nothing && isfile(p)
            push!(ids_files, parse(Int, m.captures[1]) => p)
        end
    end
    @argcheck allunique(first.(ids_files)) "Non-unique file ids, perhaps because of 0-padding?"
    sort!(ids_files, by = first)
    ids_files
end

end


### A Pluto.jl notebook ###
# v0.14.1

using Markdown
using InteractiveUtils

# ╔═╡ de098278-8e74-11eb-13e3-49c1b6e06e7d
using Pkg; Pkg.activate("MLJ_env", shared=true)

# ╔═╡ ac463e7a-8b59-11eb-229e-db560e17c5f5
begin
	using Test
	using PlutoUI
end

# ╔═╡ 8c80e072-8b59-11eb-3c21-a18fe43c4536
md"""
## Databases and SQL

ref. from book **"Data Science from Scratch"**, Chap 24

$(html"
© Pascal, April 2021
")
"""

# ╔═╡ e7373726-8b59-11eb-2a2b-b5138e4f5268
PlutoUI.TableOfContents(indent=true, depth=4, aside=true)

# ╔═╡ f5ee64b2-8b59-11eb-2751-0778efd589cd
html"""

"""

# ╔═╡ 81290d1c-8ce2-11eb-3340-337957fd81b7
html"""

"""

# ╔═╡ 8ff1bb20-8ce2-11eb-1de6-fd84daec8930
md"""
#### Create table and insert rows

"""

# ╔═╡ d3ee2138-8ce2-11eb-0b29-659a3be01512
md"""
We will represent our table in memory by a dictionary (not really space efficient).

Also, we will:
  - Support the primary key for our table (some table do not and will not have a primary key).
  - Allow for NULL value for any column of our table (but for the primary key).
"""

# ╔═╡ 6d759180-928b-11eb-1cc1-0593c7f3b0c2
begin
	const Row = Dict{Symbol, Any}
	const GRow = Union{Row, Vector{Pair{Symbol, Any}}, Vector{Any}}  ## Generic Row
	const WhereClause = Function
	const HavingClause = Function
	const S_N = Union{Symbol, Nothing}
	# const PK = Pair{S_N, DataType}
	const D_SF = Dict{Symbol, Function}
	const U_IN = Union{Int, Nothing}
	const U_SN = Union{String, Nothing}
	const UDT = Union{Union, DataType, Type};
end

# ╔═╡ 2e8684fc-2ba6-48ec-81b3-f1c31cdde43a
md"""
Let us define our custom type for a relational table:
"""

# ╔═╡ 53accaea-92d4-11eb-0bc2-3d2ee10f9bfb
mutable struct Table
	columns::Vector{Symbol}
	types::Vector{UDT}
	pkey::Union{Symbol, Nothing}
	rows::Vector{Row}

  function Table(col_types::Vector{Pair{Symbol, DT}};
               pkey::Pair=(:id => Int)) where DT <: UDT
    # vector of pairs keeps implicit order
    @assert length(col_types) ≥ 1
    #
    cols = map(((k, _)=p) -> k, col_types)
    types = map(((_, v)=p) -> v, col_types)
    cols, types = check_pk(cols, types, pkey)
    new(cols, types, pkey.first, Vector{Row}[])
  end

  function Table(cols::Vector{Symbol}, types::Vector{DT};
               pkey::Pair=(:id => Int)) where DT <: UDT
    cols, types = check_pk(cols, types, pkey)
    new(cols, types, pkey.first, Vector{Row}[])
  end
end

# ╔═╡ 1b1b8d8c-3803-43fa-b3af-7a58d75bb13b
md"""
And let us start developing our API and utilities:
"""

# ╔═╡ 6f9c58d4-92d1-11eb-2c09-cb1ea5afcd6d
begin
	import Base: length
	##
	## API for Table
	##

	id(self::Table) = self.pkey

	length(self::Table) = length(self.rows)

	function gen_pkey_value(self::Table)
	  """
	  Assuming pkey is instance of a numeric type
	  """
	  length(self.rows) == 0 && (return 1)
	  (map(r -> r[id(self)], self.rows) |> maximum)  + 1
	end

	## utility
	function check_pk(cols, types, pkey)
	  if pkey.first !== nothing
		@assert pkey[2] <: Integer "Expecting an Integer, got: $(pkey[2])"
	  end

	  if pkey.first !== nothing && pkey.first ∉ cols
		cols  = [pkey[1], cols...]
		types = [pkey[2], types...]
	  end
	  (cols, types)
	end
end

# ╔═╡ 040a7354-92d2-11eb-2825-7bac34b9fdf9
YaUsers = Table([:name => String, :num_friends => Int];
	pkey=(:user_id => Int32))

# ╔═╡ 17d214a7-be15-4236-a614-3127b535ef66
AltUsers = Table([:name => String, :num_friends => Int]) ## Default pkey

# ╔═╡ 85211d8a-be50-454d-a6c5-cd820a20f8d4
## Default pkey and NULL values allowed
AltUsers₂ = Table([:name, :num_friends], [U_SN, U_IN])

# ╔═╡ adf67244-92d4-11eb-3004-41e62e906e32
begin
	##
	## Convention: by default all tables have a primary key called :id,
	## unless explicitly stated otherwise, something like:
	##   :pkey => (:user_id, Int)
	##
	Users = Table([:name => U_SN, :num_friends => U_IN];
		pkey=(:user_id => Int))
	@assert length(Users) == 0
	Users;
end

# ╔═╡ 650237ee-928e-11eb-0e43-17ab98a0cc11
begin
	##
	## API for Table (cont'ed)
	##

	function insert(self::Table, row::Vector{Pair{Symbol, DT}}) where DT <: Any
	  """
	  row is something like [:name => "FooBar", :num_friends => 3, id(Users) => 3]
	  """
	  res = filter(p -> p.first == id(self), row)
	  has_pkey = length(res) > 0
	  ##
	  ## NULL values permitted
	  #length(row) ≠ length(self.types) && has_pkey &&
	  #  throw(ArgumentError("Mismatch with expected number of columns"))
	  ##
	  check_value_dtype(self, row)
	  ## find the pair <pkey, value> pkey is given by id(self)
	  row_ids = filter(p -> p.first == id(self), row)
	  if length(row_ids) == 0
		pk_id =  gen_pkey_value(self)
		row = [row..., id(self) => pk_id]
	  else
		id_val = row_ids[1][2]  ## value assoc with this pkey
		row_already_inserted(self, id_val) && (return nothing)
	  end
	  push!(self.rows, Dict(row...))
	end

	function insert(self::Table, row::Vector{Any})
	  """
	  row is something like: [9, "Foo", 2 ] - just pure values
	  """
	  length(row) ≠ length(self.types) && id(self) ∈ map(p -> p[1], row) &&
		throw(ArgumentError("Mismatch with expected number of columns "))
	  #
	  check_value_dtype(self, row)
	  ## row[1] is the value assoc. with pkey
	  row_already_inserted(self, row[1]) && (return nothing)
	  push!(self.rows, Dict(zip(self.columns, row)))
	end

	function insert(self::Table, row::Row)
	  """
	  row is a  Dict, like: insert(Users, Dict(:name => "Foo", :num_friends => 5))
	  """
	  length(row) ≠ length(self.types) && haskey(row, id(self)) &&
		throw(ArgumentError("Mismatch with expected number of columns"))
	  check_value_dtype(self, row)
	  if haskey(row, id(self))
		row_already_inserted(self, row[id(self)]) && (return nothing)
	  else
		row[id(self)] = gen_pkey_value(self)
	  end
	  push!(self.rows, row)
	end

	insert(self::Table, rows::Vector{T}) where T <: GRow = insert.(Ref(self), rows)

	function coltype(self::Table, colname::Symbol)::UDT
	  ix = findfirst(col -> col == colname, self.columns)
	  ix === nothing && throw(ArgumentError("column $(colname) inexistent"))
	  self.types[ix]
	end

	# function Base.show(io::IO, self::Table)
	# 	s = "num. records: $(length(self)):\n"
	# 	for rd ∈ self.rows
	# 		s₁ = []
	# 		for (k, v) ∈ rd
	# 			push!(s₁, "$(k): $(v)")
	# 		end
	# 		s = string(s, "  <", join(s₁, ", "), ">\n")
	# 	end
	# 	print(io, "$(s)")
	# end

	##
	## Internal checkers
	##

	function check_value_dtype(self::Table, values::Vector{DT}) where DT <: Any
	  dtypes = self.types
	  for (v, dt) ∈ zip(values, dtypes)
		!(typeof(v) <: dt) && v !== nothing &&
		  throw("Expected type for <$(v)>: $(dt), got: $(typeof(v))")
	  end
	end

	function check_value_dtype(self::Table,
							   row::Union{Row, Vector{Pair{Symbol, DT}}}) where DT <: Any
	  (cols, types) = self.columns, self.types
	  col_type = Dict(zip(cols, types))
	  for (k, v) ∈ row
		(haskey(col_type, k) && typeof(v) <: col_type[k]) ||
		  throw("Expected type/2: $(col_type[k]), got: $(typeof(v))")
	  end
	end

	function row_already_inserted(self::Table, id_val::Any)::Bool
		id(self) === nothing && return false  ## no pkey => ignore check
		#
		res = filter(r -> r[id(self)] == id_val, self.rows)
		length(res) > 0    ## row already inserted id > 0
	end

end

# ╔═╡ d146a8ac-92cc-11eb-29b1-bb065a468477
md"""
Insert using an array/vector:
"""

# ╔═╡ 13a0940e-8ce4-11eb-231c-f331a607203c
begin
	insert(Users, [0, "Hero", 0])
	@assert length(Users) ≥ 1
end

# ╔═╡ e3458d34-92cc-11eb-2426-79f229abd908
md"""
Insert using a vector of pairs:
"""

# ╔═╡ 96c2a668-9290-11eb-074d-19ffeb68eba6
begin
	insert(Users, [id(Users) => 1, :name => "Dunn", :num_friends => 2])
	@assert length(Users) ≥ 2
end

# ╔═╡ f1357972-92cc-11eb-1f73-43aae5a43aa5
md"""
Insert using a dictionary
"""

# ╔═╡ 5a3a8fdc-929d-11eb-25e3-e309dffdb455
begin
	## Insert with a Dict
	insert(Users, Dict(id(Users) => 2, :name => "Sue", :num_friends => 3))
	@assert length(Users) ≥ 3
end

# ╔═╡ 0292e48e-92cd-11eb-1882-81a2faaad136
md"""
Insert using a dictionary, no primary key value specified:
"""

# ╔═╡ 8545d17c-929e-11eb-0989-ffd4749bbb25
begin
	## Insert with a Dict no pkey/1
	insert(Users, Dict(:name => "Ayumi", :num_friends => 5))
	@assert length(Users) ≥ 4
end

# ╔═╡ 95d2b19e-92a4-11eb-0f91-e343c9317983
begin
	## Insert with a Dict no pkey/2
	insert(Users, [:name => "PasMas", :num_friends => 5])
	@assert length(Users) ≥ 5
end

# ╔═╡ 8a945abc-012f-4ace-a67a-b7e0f520346b
begin
	## Insert with a Dict no pkey/2 + null value (encoded as nothing)
	# insert(Users,
	# 	[:name => "Foobar", :num_friends => nothing])
	insert(Users,
		[:name => "Foobar", :num_friends => nothing, :user_id => 17])
end

# ╔═╡ 28ba0bf6-92cd-11eb-0669-75eadb768518
md"""
Insert using a collection (vector of rows):
"""

# ╔═╡ d4562f7e-9290-11eb-2d8c-952f2e0edfed
begin
	## Insert a collection (Vector of) of Records
	insert(Users, [
		[5, "Chi", 3],
		[6, "Thor", 3],
		[7, "Clive", 2],
		[8, "Devin", 2],
		[9, "Kate", 2],
		[10, "Kaze", nothing]
	])
	@assert length(Users) ≥ 10
end

# ╔═╡ f877c3e6-929c-11eb-00cd-c15568f99627
Users

# ╔═╡ d5657a88-92c4-11eb-2a4f-c7dcaa97bcf6
begin
	@test_throws ArgumentError  coltype(Users, :foobar)
	@test coltype(Users, :num_friends) == U_IN
	@test coltype(Users, :name) == U_SN
end

# ╔═╡ 80ddf0f4-8ce2-11eb-3046-331119a0dc9b
html"""

"""

# ╔═╡ 80a71408-8ce2-11eb-3978-75a4a2df9116
md"""
#### Update
"""

# ╔═╡ d1ed1e36-8ce5-11eb-34ee-4db60fb0db8a
md"""
The key features for an update:
  - what table,
  - which fields/rows
  - what their new values will be
"""

# ╔═╡ 4ccdb1ce-92c2-11eb-18dd-c510e88474c6
##
## API (cont'ed)
##
function update(self::Table, updates::Row, pred::WhereClause=row -> true)
	## 1 - Make sure key/columns and values are consistent with column types
	for (col, val) ∈ updates
		col ∉ self.columns && throw(ArgumentError("invalid column $(col)"))
		col_type = coltype(self, col)
		!(typeof(val) <: col_type) && val !== nothing &&
			throw(ArgumentError("Expected $(col_type), got $(typeof(val))"))
	end

	## 2 - OK, update
	rows_to_update = filter(((ix, r)=t) -> pred(r), collect(enumerate(self.rows)))
	for (ix, row) ∈ rows_to_update
		# self.rows[ix] = Row(row..., updates...)
		for (k, v) ∈ updates
			self.rows[ix][k] = v
		end
	end
	nothing
end

# ╔═╡ 2af6dbec-92c6-11eb-1155-596ed44b04f8
begin
	update(Users, Dict{Symbol, Any}(:num_friends => 7),
		row -> row[:name] == "Ayumi")
	Users
end

# ╔═╡ 2abb0114-92c6-11eb-01d5-cd34aa6426b2
begin
	update(Users, Dict{Symbol, Any}(:num_friends => 4), row -> row[:num_friends] == 2)
	Users
end

# ╔═╡ bc1fb8d2-92ca-11eb-14e0-c9586461bc1e
begin
	update(Users, Dict{Symbol, Any}(:num_friends => 1))
	Users
end

# ╔═╡ d2b198b6-8ce2-11eb-170e-0f17904c9f2c
html"""

"""

# ╔═╡ d37f37b8-8ce8-11eb-2c00-3f98ca407f41
md"""
#### Delete
"""

# ╔═╡ 5bf7729e-8dfd-11eb-070f-9b7ec0746bd7
##
## API (cont'ed)
##
function delete(self::Table, pred::WhereClause=row -> true)|
	self.rows = [
		row for row ∈ self.rows if !pred(row)
	]
	nothing
end

# ╔═╡ 394093fa-8cea-11eb-0071-eff3045a012b
begin
	n = length(Users)
	delete(Users, row -> row[id(Users)] == 1)
	@assert length(Users) == n - 1
end

# ╔═╡ 435c4444-8dfd-11eb-24e8-5543f905f199
begin
	delete(Users)
	@assert length(Users) == 0
end

# ╔═╡ e95983ba-8ceb-11eb-38fd-ed92cdcf754c
html"""

"""

# ╔═╡ d3a749a2-8cec-11eb-1f06-b568f244b576
md"""
#### Select
"""

# ╔═╡ 31d3226a-8cf4-11eb-0897-39989ba76b58
md"""
We will give our Table struct a select method that returns a new Table . The
method accepts two optional arguments:
  - `keep_cols` which specifies the names of the columns we want to
keep in the result. if none supply, the result contains all the columns, and

  - `add_cols` is a dictionary whose keys are new column names and whose values are functions specifying how to compute the values of the new columns.
"""

# ╔═╡ 3ec9f5d5-6d6d-4586-b68b-84e3ee5aee27
## Utility function
##
function create_res_table(self::Table, new_cols, new_types;
                          pred::Function=_a -> true)
  ## keep pkey or not...
  pred(self) ?
	Table(new_cols, new_types; pkey=id(self) => coltype(self, id(self))) :
    Table(new_cols, new_types; pkey=nothing => Nothing)
end

# ╔═╡ a56253c8-92cd-11eb-31b9-51e1fd5ae026
function select(self::Table;
		keep_cols=Vector{Symbol}[], add_cols=D_SF())::Table
	##
	length(keep_cols) == 0 && (keep_cols = self.columns)

	## New column names and types
	new_cols = [keep_cols..., collect(keys(add_cols))...]
	keep_types = [coltype(self, col) for col in keep_cols]

	## collect the rows for result table
	n_rows = Vector{Any}[]
	add_types = Any[]
	for (ix, row) ∈ enumerate(self.rows)
		n_row = Any[row[col] for col ∈ keep_cols]
		## as we process the first row, we can get the return type...
		## ...of each function defined in add_cols
		## What if no row to process => tag with Any
		for (_col_name, fn) ∈ add_cols
			r = fn(row)
			ix == 1 && push!(add_types, typeof(r))
			push!(n_row, r)
		end
		push!(n_rows, n_row)
	end

	if length(add_cols) > 0 && length(add_types) == 0
		## add as many Any as column in add_cols
		push!(add_types, repeat(Any[Any], inner=length(add_cols)))
	end

	## Create result table
	new_types = UDT[keep_types..., add_types...]
	@assert(length(new_cols) == length(new_types),
		"length(new_cols) == length(new_types)")

	n_table = create_res_table(self, new_cols, new_types;
		pred=s -> id(s) ∈ keep_cols)

	insert(n_table, n_rows)
	n_table
end

# ╔═╡ e6348294-92cb-11eb-3bfd-09d80933b33a
begin
	delete(Users)
	insert(Users, [[0, "Hero", 0],
			[1, "Dunn", 2],
			[2, "Sue", 3],
			[3, "Chi", 3],
			[4, "Thor", 3],
			[5, "Clive", 2],
			[6, "Hicks", 3],
			[7, "Devin", 2],
			[8, "Kate", 2]
	])
	insert(Users, Dict(:name => "Ayumi", :num_friends => 5))
	insert(Users, [:name => "PasMas", :num_friends => 5])
end

# ╔═╡ 96386ab8-93ef-44c7-a524-1246379d7103
Table([:user_id, :name, :num_friends], UDT[Int64, Union{Nothing, String}, Union{Nothing, Int64}])

# ╔═╡ a5441766-92cd-11eb-1c8e-edcf3db0022e
begin
	## SELECT * FROM Users;
	n₀ = length(Users)
	all_users = select(Users)
	@assert length(all_users) == n₀
end

# ╔═╡ 3bfad78b-2ebd-4067-b306-b687f5a92a10
begin
	## SELECT id FROM Users;
	user_ids = select(Users, keep_cols=[:name])
end

# ╔═╡ cfed19ac-94e2-4c84-85b4-6134765cce0b
md"""
##### Limit
"""

# ╔═╡ 386a0591-92ce-4267-b1e6-73e293eb727c
##
## API (cont'ed)
##
function limit(self::Table, num_rows::Int=5)::Table
	"""
	Only the first num_rows are returned
	"""
	@assert 1 ≤ num_rows ≤ length(self.rows) "1 ≤ $(num_rows) ≤ $(length(self.rows))"

 	n_table = create_res_table(self, self.columns, self.types;
		pred=s -> id(s) !== nothing)

	## NOTE: mark vector as GRow type
	rows = Vector{Any}[]
	for row ∈ self.rows[1:num_rows]
		push!(rows, Any[v for (_, v) ∈ row])
	end
	insert(n_table, rows)
	n_table
end

# ╔═╡ 53c15148-1b04-4d85-aac6-874a6f750a00
user_names₁ = select(Users, keep_cols=[:name]) |> limit

# ╔═╡ 1ac32ed0-6e47-4606-8b70-524159660d93
user_names₂ = select(Users, keep_cols=[:name]) |>
  u -> limit(u, 2)

# ╔═╡ be425e03-c7b4-401d-9d29-fa2b07618ee5
md"""
###### Where
"""

# ╔═╡ e99eab12-08d9-4955-ba10-2eb46f74bd85
##
## API (cont'ed)
##
function where(self::Table, pred::WhereClause=row -> true)::Table
	"""
	Only the rows that satisfy pred are returned
	"""
	n_table = create_res_table(self, self.columns, self.types;
		pred=s -> id(s) !== nothing)

	n_rows = Vector{Any}[]
	for row ∈ self.rows
		pred(row) && (push!(n_rows, [row[col] for col ∈ n_table.columns]))
	end

	insert(n_table, n_rows)
	n_table
end

# ╔═╡ b08b9470-5d53-4420-bef6-ef1c0bb4414c
begin
	dunn_ids = where(Users, row -> row[:name] == "Dunn") |>
		u -> select(u, keep_cols=[id(Users)])
	# @assert length(dunn_ids) == 1
end

# ╔═╡ 955fe2a1-22b3-44db-b23d-d595d58bac3d
begin
	amp_ids = where(Users, row -> row[:name][1] ∈ ['A', 'P']) |>
		u -> select(u, keep_cols=[id(Users), :name])
end

# ╔═╡ 964507de-c48d-4f7c-ab05-9b7739ecd5f3
begin
	ncol = length(Users.columns)
	function name_len_fn(row)::Int
		length(row[:name])
	end

	name_lengths = select(Users;
		add_cols=D_SF(:name_length => name_len_fn))
end

# ╔═╡ 13c464eb-2a59-488b-b0ab-922ecda91bbd
begin
	name_lengths₂ = select(Users;
		add_cols=Dict{Symbol, Function}(
			:name_ini => row -> string(row[:name][1]),
			:len_name => row -> length(row)
		)
	)
end

# ╔═╡ 95ae9db4-f9ba-4fde-a021-0fc952550627
md"""
##### Aside on introspection
"""

# ╔═╡ 17b2e439-4f70-4db6-a01f-638ae2f88b6f
begin
	## with explicit return type
	##
	str_fn = """
function name_len_fn(row, bar, vargs...)::UInt16
	length(row[:name])
end
"""
	expr = Meta.parse(str_fn)
end

# ╔═╡ a57b3859-d026-4325-a5b2-7d4e7b4db408
begin
	## w/o explicit return type
	##
	str_fn₂ = """
function name_len_fn(row1, bar1; foo1=10)
	length(row[:name])
end
"""
	expr₂ = Meta.parse(str_fn₂)
end

# ╔═╡ 8e25eb57-b6f0-4c61-b657-4d815f1eb31f
with_terminal() do
	dump(expr)
end

# ╔═╡ 8ea2ba50-ad35-40b2-ab2d-1c34c9d90bd9
expr.args[1], expr₂.args[1]

# ╔═╡ b34d8aaa-7696-452f-9fd7-1002aa771547
## get return type
expr.args[1].args, length(expr.args[1].args), expr.args[1].args[end]

# ╔═╡ 4ff7e8e9-354e-4c99-b6f8-aedbf7ce1f3f
expr₂.args[1].args, length(expr₂.args[1].args)

# ╔═╡ 3a438295-3d30-41cc-95bd-db3b93e62d12
function get_fn_retype(str_fn::String)::DataType
	"""
	Return output type of a user function if such type is available...
	...which is the case iff expr.args[1].args has length of 2
	Otherwise fallback to Any
	"""
	expr = Meta.parse(str_fn)
	length(expr.args[1].args) == 2 ? eval(expr.args[1].args[end]) : Any
end

# ╔═╡ 9b24fead-ccfd-43ed-afa8-ed5ef6c4cf8f
get_fn_retype(str_fn), get_fn_retype(str_fn₂)

# ╔═╡ 6e7e7896-8cf8-11eb-062e-99492ec8cff8
html"""

"""

# ╔═╡ 70f707c0-8cf6-11eb-11c2-73d7b28f7a0c
md"""
#### Group by
"""

# ╔═╡ 70d84470-8cf6-11eb-23b6-c7ba506d8552
md"""
This function will take a list of columns we want to group by and a dictionary of aggragation functions plus an optional predicate parameter (having clause) that can operate on multiple rows.
"""

# ╔═╡ c451451a-8cf7-11eb-183b-e358c9d618e0
##
## API (cont'ed)
##

function group_by(self::Table; group_by_cols::Vector{Symbol}, agg::D_SF,
	having=HavingClause=row_gp -> true)::Table
	grouped_rows = Dict{}()

	## 1 - Populate groups
	for row ∈ self.rows
		key = Tuple(row[col] for col ∈ group_by_cols)
		ary = get(grouped_rows, key, Row[])
		push!(ary, row)
		grouped_rows[key] = ary
	end

	## 2 - populate rows
	n_rows = Vector{Any}[]
	agg_types = Any[]
	for (ix, (key, rows)) ∈ enumerate(grouped_rows)
		agg_row = Any[]
		if having(rows)
			agg_row = Any[agg_row..., key...]  ## Keep Any[]
			for (_, agg_fn) ∈ agg
				r = agg_fn(rows)
				ix == 1 && push!(agg_types, typeof(r))
				push!(agg_row, r)
			end
		end
		length(agg_row) > 0 && (push!(n_rows, agg_row))
	end

	## 3 - res. table consists of group_by columns and aggregates
	new_cols = [group_by_cols..., collect(keys(agg))...] |> unique
	gp_by_types = [coltype(self, col) for col ∈ group_by_cols]
	new_types = [gp_by_types..., agg_types...]

	n_table = create_res_table(self, new_cols, new_types;
		pred=s -> id(s) ∈ new_cols)
	insert(n_table, n_rows)
	n_table
end

# ╔═╡ 30bdc788-8d01-11eb-2f8f-9fd79593ddb8
begin
	"""
	-- find number of users and smallest user_id for each possible name length:
	SELECT LENGTH(name) as name_length, MIN(user_id) AS min_user_id,
		COUNT(*) AS num_users
	FROM users
	GROUP BY LENGTH(name);
	"""
	min_user_id = rows -> minimum(row[id(Users)] for row ∈ rows)
	num_rows = rows -> length(rows)

	stats_by_len = select(Users, add_cols=D_SF(:name_length => name_len_fn)) |>
		u -> group_by(u,
				group_by_cols=[:name_length],
				agg=D_SF(:min_user_id => min_user_id, :num_users => num_rows))
	#
end

# ╔═╡ c5bac3ae-c63b-44ba-9b10-75144410f456
Users

# ╔═╡ e696c2dd-b250-418c-9161-9fd87bfc378e
begin
	"""
	-- average number of friends for users whose names start with specific letters
	-- but see only the results for letters whose corresponding average is greater
	-- than 1
	SELECT SUBSTR(name, 1, 1) AS first_letter,
		AVG(num_friends) AS avg_num_friends
	FROM users
	GROUP BY SUBSTR(name, 1, 1)
	HAVING AVG(num_friends) > 1;
	"""
	first_letter_fn = row -> row[:name] !== nothing ? string(row[:name][1]) : ""

	avg_num_friends_fn =
		rows -> sum([row[:num_friends] for row in rows]) / length(rows)

	enough_friends_fn = rows -> avg_num_friends_fn(rows) > 1.

	avg_friends_by_letter =
		select(Users, add_cols=D_SF(:first_letter => first_letter_fn)) |>
		u -> group_by(u, group_by_cols=[:first_letter],
				agg=D_SF(:avg_num_friends => avg_num_friends_fn),
				having=enough_friends_fn)
end

# ╔═╡ 84946000-8d02-11eb-3160-571efee8fb0b
html"""

"""

# ╔═╡ d05f1c40-8d17-11eb-2724-4d989c4c6b92
md"""
#### Order by
"""

# ╔═╡ 848fd812-8d02-11eb-01d6-75965b08bcc5
##
## API (cont'ed)
##
function order_by(self::Table, order::Function)::Table
	n_table = select(self)
	sort!(n_table.rows, by=order)
	n_table
end

# ╔═╡ 6cde1330-e1bb-419e-87f3-cf73c0e11e3d
friendliest_letters =  avg_friends_by_letter |>
	u -> order_by(u, row -> -row[:avg_num_friends])
# NOTE: -row(...) to reverse the sort

# ╔═╡ 7d35f85e-8e07-11eb-228c-81d28a444b52
friendliest_letters₂ =  avg_friends_by_letter |>
	u -> order_by(u, row -> -row[:avg_num_friends]) |>
	u -> limit(u, 3)
# NOTE: -row(...) to reverse the sort

# ╔═╡ 40d46802-8e0f-11eb-2e47-53096e24dbd8
html"""

"""

# ╔═╡ 80dff352-8d02-11eb-06b1-5f5e325046f5
md"""
#### Join
"""

# ╔═╡ f1af4256-8e02-11eb-0238-2515d39a89cd
##
## API (cont'ed)
##
function join(self::Table, otable::Table; left_join=false)::Table
	join_on_cols = [c for c ∈ self.columns if c ∈ otable.columns]

	add_cols = [c for c ∈ otable.columns if c ∉ join_on_cols && c != id(otable)]
	new_cols = [self.columns..., add_cols...]
	new_types = UDT[self.types..., coltype.(Ref(otable), add_cols)...]

	n_table = Table(new_cols, new_types; pkey=nothing => Nothing)

	n_rows = Vector{Any}[]
	for row ∈ self.rows
		is_join = orow -> all(orow[c] == row[c] for c ∈ join_on_cols)

		o_rows = where(otable, is_join).rows
		for o_row ∈ o_rows
			push!(n_rows, Any[[row[c] for c ∈ self.columns]...,
					[o_row[c] for c ∈ add_cols]...])
		end

		if left_join && length(o_rows) == 0
			push!(n_rows, Any[[row[c] for c ∈ self.columns]...,
					[nothing for _ ∈ add_cols]...])
		end
	end
	@show "join/4", n_rows, length(n_rows)
	insert(n_table, n_rows)
	n_table
end

# ╔═╡ 26571096-f30c-476a-ad0e-ef133ba2562f
begin
	User_Interests = Table([:user_id => Int, :interest => String]) ## Default pkey
	insert(User_Interests, [
			[1, 0, "SQL"],
			[2, 0, "NoSQL"],
			[3, 2, "SQL"],
			[4, 2, "MySQL"],
			[5, 7, "PostgreSQL"],
			[6, 7, "SQL"]
	])
end

# ╔═╡ a1a91e04-ffdb-49b1-a999-e1cf1a730cb4
begin
	sql_users = join(Users, User_Interests) |>
		u -> where(u, r -> r[:interest] == "SQL") |>
		u -> select(u, keep_cols=[:name])
end

# ╔═╡ 43a52692-8e10-11eb-2043-8f0be195f58a
begin
	sql_users₂ = join(Users, User_Interests; left_join=true) |>
		u -> where(u, r -> r[:interest] !== nothing &&
							occursin(r"SQL\z", r[:interest])) |>
		u -> select(u, keep_cols=[:name])
	@test length(sql_users₂.rows) == 6
end

# ╔═╡ Cell order:
# ╟─8c80e072-8b59-11eb-3c21-a18fe43c4536
# ╠═de098278-8e74-11eb-13e3-49c1b6e06e7d
# ╠═ac463e7a-8b59-11eb-229e-db560e17c5f5
# ╟─e7373726-8b59-11eb-2a2b-b5138e4f5268
# ╟─f5ee64b2-8b59-11eb-2751-0778efd589cd
# ╟─81290d1c-8ce2-11eb-3340-337957fd81b7
# ╟─8ff1bb20-8ce2-11eb-1de6-fd84daec8930
# ╟─d3ee2138-8ce2-11eb-0b29-659a3be01512
# ╠═6d759180-928b-11eb-1cc1-0593c7f3b0c2
# ╟─2e8684fc-2ba6-48ec-81b3-f1c31cdde43a
# ╠═53accaea-92d4-11eb-0bc2-3d2ee10f9bfb
# ╟─1b1b8d8c-3803-43fa-b3af-7a58d75bb13b
# ╠═6f9c58d4-92d1-11eb-2c09-cb1ea5afcd6d
# ╠═040a7354-92d2-11eb-2825-7bac34b9fdf9
# ╠═17d214a7-be15-4236-a614-3127b535ef66
# ╠═85211d8a-be50-454d-a6c5-cd820a20f8d4
# ╠═adf67244-92d4-11eb-3004-41e62e906e32
# ╠═650237ee-928e-11eb-0e43-17ab98a0cc11
# ╟─d146a8ac-92cc-11eb-29b1-bb065a468477
# ╠═13a0940e-8ce4-11eb-231c-f331a607203c
# ╟─e3458d34-92cc-11eb-2426-79f229abd908
# ╠═96c2a668-9290-11eb-074d-19ffeb68eba6
# ╟─f1357972-92cc-11eb-1f73-43aae5a43aa5
# ╠═5a3a8fdc-929d-11eb-25e3-e309dffdb455
# ╟─0292e48e-92cd-11eb-1882-81a2faaad136
# ╠═8545d17c-929e-11eb-0989-ffd4749bbb25
# ╠═95d2b19e-92a4-11eb-0f91-e343c9317983
# ╠═8a945abc-012f-4ace-a67a-b7e0f520346b
# ╟─28ba0bf6-92cd-11eb-0669-75eadb768518
# ╠═d4562f7e-9290-11eb-2d8c-952f2e0edfed
# ╠═f877c3e6-929c-11eb-00cd-c15568f99627
# ╠═d5657a88-92c4-11eb-2a4f-c7dcaa97bcf6
# ╟─80ddf0f4-8ce2-11eb-3046-331119a0dc9b
# ╟─80a71408-8ce2-11eb-3978-75a4a2df9116
# ╟─d1ed1e36-8ce5-11eb-34ee-4db60fb0db8a
# ╠═4ccdb1ce-92c2-11eb-18dd-c510e88474c6
# ╠═2af6dbec-92c6-11eb-1155-596ed44b04f8
# ╠═2abb0114-92c6-11eb-01d5-cd34aa6426b2
# ╠═bc1fb8d2-92ca-11eb-14e0-c9586461bc1e
# ╟─d2b198b6-8ce2-11eb-170e-0f17904c9f2c
# ╟─d37f37b8-8ce8-11eb-2c00-3f98ca407f41
# ╠═5bf7729e-8dfd-11eb-070f-9b7ec0746bd7
# ╠═394093fa-8cea-11eb-0071-eff3045a012b
# ╠═435c4444-8dfd-11eb-24e8-5543f905f199
# ╟─e95983ba-8ceb-11eb-38fd-ed92cdcf754c
# ╟─d3a749a2-8cec-11eb-1f06-b568f244b576
# ╟─31d3226a-8cf4-11eb-0897-39989ba76b58
# ╠═3ec9f5d5-6d6d-4586-b68b-84e3ee5aee27
# ╠═a56253c8-92cd-11eb-31b9-51e1fd5ae026
# ╠═e6348294-92cb-11eb-3bfd-09d80933b33a
# ╠═96386ab8-93ef-44c7-a524-1246379d7103
# ╠═a5441766-92cd-11eb-1c8e-edcf3db0022e
# ╠═3bfad78b-2ebd-4067-b306-b687f5a92a10
# ╟─cfed19ac-94e2-4c84-85b4-6134765cce0b
# ╠═386a0591-92ce-4267-b1e6-73e293eb727c
# ╠═53c15148-1b04-4d85-aac6-874a6f750a00
# ╠═1ac32ed0-6e47-4606-8b70-524159660d93
# ╟─be425e03-c7b4-401d-9d29-fa2b07618ee5
# ╠═e99eab12-08d9-4955-ba10-2eb46f74bd85
# ╠═b08b9470-5d53-4420-bef6-ef1c0bb4414c
# ╠═955fe2a1-22b3-44db-b23d-d595d58bac3d
# ╠═964507de-c48d-4f7c-ab05-9b7739ecd5f3
# ╠═13c464eb-2a59-488b-b0ab-922ecda91bbd
# ╟─95ae9db4-f9ba-4fde-a021-0fc952550627
# ╠═17b2e439-4f70-4db6-a01f-638ae2f88b6f
# ╠═a57b3859-d026-4325-a5b2-7d4e7b4db408
# ╠═8e25eb57-b6f0-4c61-b657-4d815f1eb31f
# ╠═8ea2ba50-ad35-40b2-ab2d-1c34c9d90bd9
# ╠═b34d8aaa-7696-452f-9fd7-1002aa771547
# ╠═4ff7e8e9-354e-4c99-b6f8-aedbf7ce1f3f
# ╠═3a438295-3d30-41cc-95bd-db3b93e62d12
# ╠═9b24fead-ccfd-43ed-afa8-ed5ef6c4cf8f
# ╟─6e7e7896-8cf8-11eb-062e-99492ec8cff8
# ╟─70f707c0-8cf6-11eb-11c2-73d7b28f7a0c
# ╟─70d84470-8cf6-11eb-23b6-c7ba506d8552
# ╠═c451451a-8cf7-11eb-183b-e358c9d618e0
# ╠═30bdc788-8d01-11eb-2f8f-9fd79593ddb8
# ╠═c5bac3ae-c63b-44ba-9b10-75144410f456
# ╠═e696c2dd-b250-418c-9161-9fd87bfc378e
# ╟─84946000-8d02-11eb-3160-571efee8fb0b
# ╟─d05f1c40-8d17-11eb-2724-4d989c4c6b92
# ╠═848fd812-8d02-11eb-01d6-75965b08bcc5
# ╠═6cde1330-e1bb-419e-87f3-cf73c0e11e3d
# ╠═7d35f85e-8e07-11eb-228c-81d28a444b52
# ╟─40d46802-8e0f-11eb-2e47-53096e24dbd8
# ╟─80dff352-8d02-11eb-06b1-5f5e325046f5
# ╠═f1af4256-8e02-11eb-0238-2515d39a89cd
# ╠═26571096-f30c-476a-ad0e-ef133ba2562f
# ╠═a1a91e04-ffdb-49b1-a999-e1cf1a730cb4
# ╠═43a52692-8e10-11eb-2043-8f0be195f58a


#=
Created on 20/01/2021 16:45:00
Last update: -

@author: Michiel Stock
michielfmstock@gmail.com

Illustration of the composite and parametric types:
- composite types
    - mutable/immuatble
- unions
- Parametric types
=#

#=
Composite types, sometimes call records, structs or object, can store several values in its *fields*.

When defining a new composite type, we can choose them to be mutable or immuatble:
- mutable types are defined using `mutable struct ... end`, they allow the fields to be changed after the object is created;
- immuatble types are defined similarly using `mutable struct ... end`, after creating the object its fields cannot be changed.
Mutable types are a bit more flexible, though might be a somewhat less safe and are more difficult to work with. As the compiler
knows everything in advance, it might better optimize for immutable types. Which one you choose depends on your application, though
generally immuatable types are the better choice!

As an example, let us define an agent type for an ecological individual-based model (IBM). We create the abstract type `Agent` for which we
can then specify several children types.
=#

abstract type Agent end

#=
The concrete types in such an IBM might represent an animal type you want to model, for example preys and predators. Making a concrete type of
a prey animal, we want each to have an unique identifier (represented by an integer) and a position. As we expect the agent to move, hence changing
its position when our simulation runs, we choose a mutable type.
=#

mutable struct Prey <: Agent
    id::Int
    pos
end
# notice the type annotation for `id`, which we choose to always reprsenent by an integer.
# the position might be represented by (x,y) coordinates, or as a location ID, or a position on a grid. We don't know, so we leave it untyped.

# defining a composite type immediately a constructor available.
deer = Prey(1, (0.5, 1.9))

# you can always check which field names are available
fieldnames(Prey)

# the fields can be accessed simply
deer.id, deer.pos

# FYI: This is just syntactic sugar for `getfield`, e.g. `getfield(deer, :id)`

# Similarly, a predator type can be defined. In addition to an id and position, which each agent has, they also have a size, determining its mobility.

mutable struct Predator <: Agent
    id::Int
    pos
    size::Float64
end

wolf = Predator(2, (0.0, 0.0), 40.0)  # 40 kg wolf

#=
Using the `.` syntax for accessing the fields is not very tidy! We should define custom getter functions
for the user to access the relevant fields. We could define `id` and `position` methods to get the respective
fields for the two agents. However, since these fields should be defined for every `Agent` type, we can just create
these for the Agent type!
=#

id(agent::Agent) = agent.id
position(agent::Agent) = agent.pos

# Here, we could theoretically have ommited the type annotation in the function. Then the function would accept
# objects of the non-agent type and likely yield an error because they don't have the `id` or `pos` field. Now,
# these functions will return a `MethodError` when given a non-`Agent` input.

# A slightly more interesting example is by extending `size`.

Base.size(agent::Predator) = agent.size

# Here, we had to import `size` because we are extending a function from the `Base` library to work with a new type (doing something vastly different
# than its original function).

# similarly, we can program behaviour between the agents

interact(agent1::Agent, agent2::Agent) = nothing
interact(agent1::Predator, agent2::Prey) = "eat"
interact(agent1::Prey, agent2::Predator) = "run"

# We have chosen the default behaviour that two Agents of unspecified types do not interact at al,
# this will now be the case when a prey meets other prey, a predator an other predator or a new third type comes into the equation.

# FYI: Since in these simple examples, the `interact` methods do not use their arguments, merely perform type checking, we could have written this as `interact(::Agent,::Agent) = ...` etc.

# PARAMETRIC TYPES

#=
Sometimes we want more flexiblility in defining types. Think of designing a new type of matrix. Here you would like to work them for all
numeric datatypes, Int, Int8, Float6, Rational, in addition to new datatypes that might not even be defined yet! To this end, we use
*parametric types*, types that depend on another type.

For example, consider a 2-dimensional coordinate:
=#

struct Point{T}
    x::T
    y::T
end

# Here, each coordinate of the type `Point` has two attributes, `x` and `y`, of the same type. The specific type of Point can vary.

p = Point(1.0, 2.0)

Point(1, 2)

# note that

p isa Point

# But what will happen if you evaluate `Point(1, 2.0)`?

Point(1, 2.0)

# Parametric types can be used in dispatch. For example, if we want to compute the norm of a Point, this would only make sense if Point is a number.

norm(p::Point{T} where {T<:Number}) = sqrt(p.x^2 + p.y^2)

norm(p)

# Constructors

## Outer constructors

# Constructors are functions that create new objects. We have already seen that when creating a new `struc`, this immediately initiates the constructor (e.g., `Point(1.0, 2.0)`). These can also be made explicitly:

Point(x::T, y::T) where {T<:Real} = Point{T}(x,y)

# Constructors, however, allow us to have custom behavior when initializing types. For example, we have seen that `Point(1, 2.0)` won't work, because the two inputs are of the same type.
# In this case, we can make the rule that one of the inputs has to be promoted to a more general type.

Point(x::Real, y::Real) = Point(promote(x, y)...)

Point(1, 2.0)

# We can write other constructors just like functions. For example, support that when we provide a single `x`, we want to create a point (x, y):

Point(x) = Point(x, x);

Point(1)

## Inner constructors

#The above examples show *outer constructors*. These are defined outside the structure. We can also use *inner constructors*, which are declared within the definition of the type. These make use of the keyword `new`. For example, let us define an ordered pair.

struct OrderedPair
  x
  y
  function OrderedPair(x, y)
    if x < y
      new(x, y)
    else
      new(y, x)
    end
  end
end

OrderedPair(18, 23)

OrderedPair(8, 2)

# FYI: for parametric types, the `new` keyword should be type annotated. So, for in the `Point` example one would use `new{T}(x,y)`.


"""
    ispangram(input)

Return `true` if `input` contains every alphabetic character (case insensitive).

"""
function ispangram(input)
    input_low = lowercase(input)
    lower_alpha = 'a':'z'

    return all(
        [(c in input_low) for c in lower_alpha])

end


import Base.BaseDocs: @kw_str

"""
**欢迎来到 Julia $(string(VERSION)).** 完整的中文手册可以在这里找到

    https://docs.juliacn.com/

更多中文资料和教程，也请关注 Julia 中文社区

    https://cn.julialang.org

新手请参考中文 discourse 上的新手指引

    https://discourse.juliacn.com/t/topic/159

输入 `?`， 然后输入你想要查看帮助文档的函数或者宏名称就可以查看它们的文档。例如 `?cos`, 或者 `?@time` 然后按回车键即可。

在 REPL 中输入 `ENV["REPL_LOCALE"]=""` 将恢复英文模式。再次回到中文模式请输入 `ENV["REPL_LOCALE"]="zh_CN"`。
"""
kw"help", kw"?", kw"julia", kw""

"""
    using

`using Foo` 将会加载一个名为 `Foo` 的模块（module）或者一个包，然后其 [`export`](@ref) 的名称将可以直接使用。不论是否被 `export`，名称都可以通过点来访问（例如，输入 `Foo.foo` 来访问到 `foo`）。查看[手册中关于模块的部分](@ref modules)以获取更多细节。
"""
kw"using"

"""
    import

`import Foo` 将会加载一个名为 `Foo` 的模块（module）或者一个包。`Foo` 模块中的名称可以通过点来访问到（例如，输入 `Foo.foo` 可以获取到 `foo`）。查看[手册中关于模块的部分](@ref modules)以获取更多细节。
"""
kw"import"

"""
    export

`export` 被用来在模块中告诉Julia哪些函数或者名字可以由用户使用。例如 `export foo` 将在 [`using`](@ref) 这个 module 的时候使得 `foo`可以直接被访问到。查看[手册中关于模块的部分](@ref modules)以获取更多细节。
"""
kw"export"

"""
    abstract type

`abstract type` 声明来一个不能实例化的类型，它将仅仅作为类型图中的一个节点存在，从而能够描述一系列相互关联的具体类型（concrete type）：这些具体类型都是抽象类型的子节点。抽象类型在概念上使得 Julia 的类型系统不仅仅是一系列对象的集合。例如：

```julia
abstract type Number end
abstract type Real <: Number end
```

[`Number`](@ref) 没有父节点（父类型）, 而 [`Real`](@ref) 是 `Number` 的一个抽象子类型。
"""
kw"abstract type"

"""
    module

`module` 会声明一个 `Module` 类型的实例用于描述一个独立的变量名空间。在一个模块（module）里，你可以控制来自于其它模块的名字是否可见（通过载入，import），你也可以决定你的名字有哪些是可以公开的（通过暴露，export）。模块使得你在在创建上层定义时无需担心命名冲突。查看[手册中关于模块的部分](@ref modules)以获取更多细节。

# 例子
```julia
module Foo
import Base.show
export MyType, foo

struct MyType
    x
end

bar(x) = 2x
foo(a::MyType) = bar(a.x) + 1
show(io::IO, a::MyType) = print(io, "MyType \$(a.x)")
end
```
"""
kw"module"

"""
    baremodule

`baremodule` 将声明一个不包含 `using Base` 或者 `eval` 定义的模块。但是它将仍然载入 `Core` 模块。
"""
kw"baremodule"

"""
    primitive type

`primitive type` 声明了一个其数据仅仅由一系列二进制数表示的具体类型。比较常见的例子是整数类型和浮点类型。下面是一些内置的原始类型（primitive type）：

```julia
primitive type Char 32 end
primitive type Bool <: Integer 8 end
```

名称后面的数字表达了这个类型存储所需的比特数目。目前这个数字要求是 8 bit 的倍数。[`Bool`](@ref) 类型的声明展示了一个原始类型如何选择成为另一个类型的子类型。
"""
kw"primitive type"

"""
    macro

`macro` 定义了一种会将生成的代码包含在最终程序体中的方法，这称之为宏。一个宏将一系列输入映射到一个表达式，然后所返回的表达式将会被直接进行编译而不需要在运行时调用 `eval` 函数。宏的输入可以包括表达式、字面量和符号。例如：

# 例子

```jldoctest
julia> macro sayhello(name)
           return :( println("Hello, ", \$name, "!") )
       end
@sayhello (macro with 1 method)

julia> @sayhello "小明"
Hello, 小明!
```
"""
kw"macro"

"""
    local

`local`将会定义一个新的局部变量。

查看[手册：变量作用域](@ref scope-of-variables)以获取更详细的信息。

# 例子
```jldoctest
julia> function foo(n)
           x = 0
           for i = 1:n
               local x # introduce a loop-local x
               x = i
           end
           x
       end
foo (generic function with 1 method)

julia> foo(10)
0
```
"""
kw"local"

"""
    global

`global x` 将会使得当前作用域和当前作用所包含的作用域里的 `x` 指向名为 `x` 的全局变量。查看[手册：变量作用域](@ref scope-of-variables)以获取更多信息。

# 例子
```jldoctest
julia> z = 3
3

julia> function foo()
           global z = 6 # use the z variable defined outside foo
       end
foo (generic function with 1 method)

julia> foo()
6

julia> z
6
```
"""
kw"global"

"""
    let

`let` 会在每次被运行时声明一个新的变量绑定。这个新的变量绑定将拥有一个新的地址。这里的不同只有当变量通过闭包生存在它们的作用域外时才会显现。`let` 语法接受逗号分割的一系列赋值语句和变量名：

```julia
let var1 = value1, var2, var3 = value3
    code
end
```

这些赋值语句是按照顺序求值的，等号右边的表达式将会首先求值，然后才绑定给左边的变量。因此这使得 `let x = x` 这样的表达式有意义，因为这两个 `x` 变量将具有不同的地址。
"""
kw"let"

"""
    quote

`quote` 会将其包含的代码扩变成一个多重的表达式对象，而无需显示调用 `Expr` 的构造器。这称之为引用，比如说

```julia
ex = quote
    x = 1
    y = 2
    x + y
end
```

和其它引用方式不同的是，`:( ... )`形式的引用（被包含时）将会在表达式树里引入一个在操作表达式树时必须要考虑的 `QuoteNode` 元素。而在其它场景下，`:( ... )`和 `quote .. end` 代码块是被同等对待的。
"""
kw"quote"

"""
    '

厄米算符（共轭转置），参见 [`adjoint`](@ref)

# 例子
```jldoctest
julia> A = [1.0 -2.0im; 4.0im 2.0]
2×2 Array{Complex{Float64},2}:
 1.0+0.0im  -0.0-2.0im
 0.0+4.0im   2.0+0.0im

julia> A'
2×2 Array{Complex{Float64},2}:
  1.0-0.0im  0.0-4.0im
 -0.0+2.0im  2.0-0.0im
```
"""
kw"'"

"""
    const

`const` 被用来声明常数全局变量。在大部分（尤其是性能敏感的代码）全局变量应当被声明为常数。

```julia
const x = 5
```

可以使用单个 `const` 声明多个常数变量。

```julia
const y, z = 7, 11
```

注意 `const` 只会作用于一个 `=` 操作，因此 `const x = y = 1` 声明了 `x` 是常数，而 `y` 不是。在另一方面，`const x = const y = 1`声明了 `x` 和 `y` 都是常数。

注意「常数性质」并不会强制容器内部变成常数，所以如果 `x` 是一个数组或者字典（举例来讲）你仍然可以给它们添加或者删除元素。

严格来讲，你甚至可以重新定义 `const`（常数）变量，尽管这将会让编译器产生一个警告。唯一严格的要求是这个变量的**类型**不能改变，这也是为什么常数变量会比一般的全局变量更快的原因。
"""
kw"const"

"""
    function

函数由 `function` 关键词定义：

```julia
function add(a, b)
    return a + b
end
```

或者是更短的形式：

```julia
add(a, b) = a + b
```

[`return`](@ref) 关键词的使用方法和其它语言完全一样，但是常常是不使用的。一个没有显示声明 `return` 的函数将返回函数体最后一个表达式。
"""
kw"function"

"""
    return

`return` 可以用来在函数体中立即退出并返回给定值，例如

```julia
function compare(a, b)
    a == b && return "equal to"
    a < b ? "less than" : "greater than"
end
```

通常，你可以在函数体的任意位置放置 `return` 语句，包括在多层嵌套的循环和条件表达式中，但要注意 `do` 块。例如：

```julia
function test1(xs)
    for x in xs
        iseven(x) && return 2x
    end
end

function test2(xs)
    map(xs) do x
        iseven(x) && return 2x
        x
    end
end
```

在第一个例子中，return 一碰到偶数就跳出包含它的函数，因此 `test1([5,6,7])` 返回 `12`。

你可能希望第二个例子的行为与此相同，但实际上，这里的 `return` 只会跳出（在 `do` 块中的）*内部*函数并把值返回给 `map`。于是，`test2([5,6,7])` 返回 `[5,12,7]`。
"""
kw"return"

# 仿照 https://docs.juliacn.com/latest/manual/control-flow/#man-conditional-evaluation-1
"""
    if/elseif/else

`if`/`elseif`/`else` 执行条件表达式（Conditional evaluation）可以根据布尔表达式的值，让部分代码被执行或者不被执行。下面是对 `if`-`elseif`-`else` 条件语法的分析：

```julia
if x < y
    println("x is less than y")
elseif x > y
    println("x is greater than y")
else
    println("x is equal to y")
end
```

如果表达式 `x < y` 是 `true`，那么对应的代码块会被执行；否则判断条件表达式 `x > y`，如果它是 `true`，则执行对应的代码块；如果没有表达式是 true，则执行 `else` 代码块。`elseif` 和 `else` 代码块是可选的，并且可以使用任意多个 `elseif` 代码块。
"""
kw"if", kw"elseif", kw"else"

"""
    for

`for` 循环通过迭代一系列值来重复计算循环体。

# 例子
```jldoctest
julia> for i in [1, 4, 0]
           println(i)
       end
1
4
0
```
"""
kw"for"

# 仿照 https://docs.juliacn.com/latest/manual/control-flow/#man-loops-1
"""
    while

`while` 循环会重复执行条件表达式，并在该表达式为 `true` 时继续执行 while 循环的主体部分。当 while 循环第一次执行时，如果条件表达式为 false，那么主体代码就一次也不会被执行。

# 例子
jldoctest
julia> i = 1
1

julia> while i < 5
           println(i)
           global i += 1
       end
1
2
3
4
```
"""
kw"while"

"""
    end

`end` 标记一个表达式块的结束，例如 [`module`](@ref)、[`struct`](@ref)、[`mutable struct`](@ref)、[`begin`](@ref)、[`let`](@ref)、[`for`](@ref) 等。`end` 在索引数组时也可以用来表示维度的最后一个索引。

# 例子
```jldoctest
julia> A = [1 2; 3 4]
2×2 Array{Int64,2}:
 1  2
 3  4

julia> A[end, :]
2-element Array{Int64,1}:
 3
 4
```
"""
kw"end"

# 仿照 https://docs.juliacn.com/latest/manual/control-flow/#try/catch-%E8%AF%AD%E5%8F%A5-1
"""
    try/catch

`try/catch` 语句可以用来捕获 `Exception`，并进行异常处理。例如，一个自定义的平方根函数可以通过 `Exception` 来实现自动按需调用求解实数或者复数平方根的方法：

```julia
f(x) = try
    sqrt(x)
catch
    sqrt(complex(x, 0))
end
```

`try/catch` 语句允许保存 `Exception` 到一个变量中，例如 `catch y`。

`try/catch` 组件的强大之处在于能够将高度嵌套的计算立刻解耦成更高层次地调用函数。
"""
kw"try", kw"catch"

# 仿照 https://docs.juliacn.com/latest/manual/control-flow/#finally-%E5%AD%90%E5%8F%A5-1
"""
    finally

无论代码块是如何退出的，都让代码块在退出时运行某段代码。这里是一个确保一个打开的文件被关闭的例子：

```julia
f = open("file")
try
    operate_on_file(f)
finally
    close(f)
end
```

当控制流离开 `try` 代码块（例如，遇到 `return`，或者正常结束），`close(f)` 就会被执行。如果 `try` 代码块由于异常退出，这个异常会继续传递。`catch` 代码块可以和 `try` 还有 `finally` 配合使用。这时 `finally` 代码块会在 `catch` 处理错误之后才运行。
"""
kw"finally"

"""
    break

立即跳出当前循环。

# 例子
```jldoctest
julia> i = 0
0

julia> while true
           global i += 1
           i > 5 && break
           println(i)
       end
1
2
3
4
5
```
"""
kw"break"

"""
    continue

跳过当前循环迭代的剩余部分。

# 例子
```jldoctest
julia> for i = 1:6
           iseven(i) && continue
           println(i)
       end
1
3
5
```
"""
kw"continue"

"""
    do

创建一个匿名函数。例如：

```julia
map(1:10) do x
    2x
end
```

等价于 `map(x->2x, 1:10)`。

像这样便可使用多个参数：

```julia
map(1:10, 11:20) do x, y
    x + y
end
```
"""
kw"do"

"""
    ...

「splat」运算符 `...` 表示参数序列。`...` 可以在函数定义中用来表示该函数接受任意数量的参数。`...` 也可以用来将函数作用于参数序列。

# 例子
```jldoctest
julia> add(xs...) = reduce(+, xs)
add (generic function with 1 method)

julia> add(1, 2, 3, 4, 5)
15

julia> add([1, 2, 3]...)
6

julia> add(7, 1:100..., 1000:1100...)
111107
```
"""
kw"..."

"""
    ;

`;` 在 Julia 中具有与许多类 C 语言相似的作用，用于分隔前一个语句的结尾。`;` 在换行中不是必要的，但可以用于在单行中分隔语句或者将多个表达式连接为单个表达式。`;` 也用于抑制 REPL 和类似界面中的输出打印。

# 例子
```julia
julia> function foo()
           x = "Hello, "; x *= "World!"
           return x
       end
foo (generic function with 1 method)

julia> bar() = (x = "Hello, Mars!"; return x)
bar (generic function with 1 method)

julia> foo();

julia> bar()
"Hello, Mars!"
```
"""
kw";"

"""
    x && y

短路布尔 AND。
"""
kw"&&"

"""
    x || y

短路布尔 OR。
"""
kw"||"

"""
    ccall((function_name, library), returntype, (argtype1, ...), argvalue1, ...)
    ccall(function_name, returntype, (argtype1, ...), argvalue1, ...)
    ccall(function_pointer, returntype, (argtype1, ...), argvalue1, ...)

调用由 C 导出的共享库里的函数，该函数由元组 `(function_name, library)` 指定，其中的每个组件都是字符串或符号。若不指定库，还可以使用 `function_name` 的符号或字符串，它会在当前进程中解析。另外，`ccall` 也可用于调用函数指针 `function_pointer`，比如 `dlsym` 返回的函数指针。

请注意参数类型元组必须是字面上的元组，而不是元组类型的变量或表达式。

通过自动插入对 `unsafe_convert(argtype, cconvert(argtype, argvalue))` 的调用，每个传给 `ccall` 的 `argvalue` 将被类型转换为对应的 `argtype`。（有关的详细信息，请参阅 [`unsafe_convert`](@ref Base.unsafe_convert) 和 [`cconvert`](@ref Base.cconvert) 的文档。）在大多数情况下，这只会简单地调用 `convert(argtype, argvalue)`。
"""
kw"ccall"

"""
    begin

`begin...end` 表示一个代码块。

```julia
begin
    println("Hello, ")
    println("World!")
end
```

通常，`begin` 不会是必需的，因为诸如 [`function`](@ref) and [`let`](@ref) 之类的关键字会隐式地开始代码块。另请参阅 [`;`](@ref)。
"""
kw"begin"

"""
    struct

struct 是 Julia 中最常用的数据类型，由名称和一组字段指定。

```julia
struct Point
    x
    y
end
```

可对字段施加类型限制，该限制也可被参数化：

```julia
    struct Point{X}
        x::X
        y::Float64
    end
```

struct 可以通过 `<:` 语法声明一个抽象超类型：
A struct can also declare an abstract super type via `<:` syntax:

```julia
struct Point <: AbstractPoint
    x
    y
end
```

`struct` 默认是不可变的；这些类型的实例在构造后不能被修改。如需修改实例，请使用 [`mutable struct`](@ref) 来声明一个可以修改其实例的类型。

有关更多细节，比如怎么定义构造函数，请参阅手册的 [复合类型](@ref) 章节。
"""
kw"struct"

"""
    mutable struct

`mutable struct` 类似于 [`struct`](@ref)，但另外允许在构造后设置类型的字段。有关详细信息，请参阅 [复合类型](@ref)。
"""
kw"mutable struct"

"""
    new

仅在内部构造函数中可用的特殊函数，用来创建该类型的对象。有关更多信息，请参阅手册的 [内部构造方法](@ref) 章节。
"""
kw"new"

"""
    where

`where` 关键字创建一个类型，该类型是其他类型在一些变量上所有值的迭代并集。例如 `Vector{T} where T<:Real` 包含所有元素类型是某种 `Real` 的 [`Vector`](@ref)。

如果省略，变量上界默认为 `Any`：

```julia
Vector{T} where T    # short for `where T<:Any`
```

变量也可以具有下界：

```julia
Vector{T} where T>:Int
Vector{T} where Int<:T<:Real
```

嵌套的 `where` 也有简洁的语法。例如，这行代码：

```julia
Pair{T, S} where S<:Array{T} where T<:Number
```

可以缩写为：

```julia
Pair{T, S} where {T<:Number, S<:Array{T}}
```

这种形式常见于方法签名：

请注意，在这种形式中，最外层变量列在最前面。这与使用语法 `T{p1, p2, ...}` 将类型「作用」于参数值时所替换变量的次序相匹配。
"""
kw"where"

"""
    ans

一个引用最后一次计算结果的变量，在交互式提示符中会自动设置。
"""
kw"ans"

"""
    Union{}

`Union{}`，即空的类型 [`Union`](@ref)，是没有值的类型。也就是说，它具有决定性性质：对于任何 `x`，`isa(x, Union{}) == false`。`Base.Bottom` 被定义为其别名，`Union{}` 的类型是 `Core.TypeofBottom`。

# 例子
```jldoctest
julia> isa(nothing, Union{})
false
```
"""
kw"Union{}", Base.Bottom

"""
    ::

`::` 运算符用于类型声明，在程序中可被附加到表达式和变量后。详见手册的 [类型声明](@ref) 章节

在类型声明外，`::` 用于断言程序中的表达式和变量具有给定类型。

# 例子
```jldoctest
julia> (1+2)::AbstractFloat
ERROR: TypeError: typeassert: expected AbstractFloat, got Int64

julia> (1+2)::Int
3
```
"""
kw"::"

"""
Julia 的基础库。
"""
kw"Base"


module Example

export func

"""
    func(x)

Returns double the number `x` plus `1`.

Here's an equation:

``\\frac{n!}{k!(n - k)!} = \\binom{n}{k}``

"""
func(x) = 2x + 1

end


#### Elementwise manipulations (scaling/clamping/type conversion) ####

# This file exists primarily to handle conversions for display and
# saving to disk. Both of these operations require UFixed-valued
# elements, but with display we always want to convert to 8-bit
# whereas saving can handle 16-bit.
# We also can't trust that user images are clamped properly.
# Finally, this supports adjustable contrast limits.

# Structure of MapInfo subtype definitions:
#   - type definition
#   - constructors for scalars
#   - constructors for AbstractArrays
#   - similar  (syntax: similar(mapi, ToType, FromType))
#   - implementation of map() for scalars
#   - implementation of map() for AbstractArrays
# map(mapi::MapInfo{T}, x) should return an object of type T (for x not an array)
# map1(mapi::MapInfo{T}, x) is designed to allow T<:Color to work on
#    scalars x::Fractional

# Dispatch-based elementwise manipulations
"""
`MapInfo{T}` is an abstract type that encompasses objects designed to
perform intensity or color transformations on pixels.  For example,
before displaying an image in a window, you might need to adjust the
contrast settings; `MapInfo` objects provide a means to describe these
transformations without calculating them immediately.  This delayed
execution can be useful in many contexts.  For example, if you want to
display a movie, it would be quite wasteful to have to first transform
the entire movie; instead, `MapInfo` objects allow one to specify a
transformation to be performed on-the-fly as particular frames are
displayed.

You can create your own custom `MapInfo` objects. For example, given a
grayscale image, you could color "saturated" pixels red using

```jl
immutable ColorSaturated{C<:AbstractRGB} <: MapInfo{C}
end

Base.map{C}(::ColorSaturated{C}, val::Union{Number,Gray}) = ifelse(val == 1, C(1,0,0), C(val,val,val))

imgc = map(ColorSaturated{RGB{U8}}(), img)
```

For pre-defined types see `MapNone`, `BitShift`, `ClampMinMax`, `ScaleMinMax`,
`ScaleAutoMinMax`, and `ScaleSigned`.
"""
abstract MapInfo{T}
eltype{T}(mapi::MapInfo{T}) = T

## MapNone
"`MapNone(T)` is a `MapInfo` object that converts `x` to have type `T`."
immutable MapNone{T} <: MapInfo{T}; end

# Constructors
MapNone{T}(::Type{T}) = MapNone{T}()
MapNone{T}(val::T) = MapNone{T}()
MapNone{T}(A::AbstractArray{T}) = MapNone{T}()

similar{T}(mapi::MapNone, ::Type{T}, ::Type) = MapNone{T}()

# Implementation
map{T}(mapi::MapNone{T}, val::Union{Number,Colorant}) = convert(T, val)
map1(mapi::Union{MapNone{RGB24}, MapNone{ARGB32}}, b::Bool) = ifelse(b, 0xffuf8, 0x00uf8)
map1(mapi::Union{MapNone{RGB24},MapNone{ARGB32}}, val::Fractional) = convert(UFixed8, val)
map1{CT<:Colorant}(mapi::MapNone{CT}, val::Fractional) = convert(eltype(CT), val)

map{T<:Colorant}(mapi::MapNone{T}, img::AbstractImageIndexed{T}) = convert(Image{T}, img)
map{C<:Colorant}(mapi::MapNone{C}, img::AbstractImageDirect{C}) = img  # ambiguity resolution
map{T}(mapi::MapNone{T}, img::AbstractArray{T}) = img

## BitShift
"""
`BitShift{T,N}` performs a "saturating rightward bit-shift" operation.
It is particularly useful in converting high bit-depth images to 8-bit
images for the purpose of display.  For example,

```
map(BitShift(UFixed8, 8), 0xa2d5uf16) === 0xa2uf8
```

converts a `UFixed16` to the corresponding `UFixed8` by discarding the
least significant byte.  However,

```
map(BitShift(UFixed8, 7), 0xa2d5uf16) == 0xffuf8
```

because `0xa2d5>>7 == 0x0145 > typemax(UInt8)`.

When applicable, the main advantage of using `BitShift` rather than
`MapNone` or `ScaleMinMax` is speed.
"""
immutable BitShift{T,N} <: MapInfo{T} end
BitShift{T}(::Type{T}, n::Int) = BitShift{T,n}()  # note that this is not type-stable

similar{S,T,N}(mapi::BitShift{S,N}, ::Type{T}, ::Type) = BitShift{T,N}()

# Implementation
immutable BS{N} end
_map{T<:Unsigned,N}(::Type{T}, ::Type{BS{N}}, val::Unsigned) = (v = val>>>N; tm = oftype(val, typemax(T)); convert(T, ifelse(v > tm, tm, v)))
_map{T<:UFixed,N}(::Type{T}, ::Type{BS{N}}, val::UFixed) = reinterpret(T, _map(FixedPointNumbers.rawtype(T), BS{N}, reinterpret(val)))
map{T<:Real,N}(mapi::BitShift{T,N}, val::Real) = _map(T, BS{N}, val)
map{T<:Real,N}(mapi::BitShift{Gray{T},N}, val::Gray) = Gray(_map(T, BS{N}, val.val))
map1{N}(mapi::Union{BitShift{RGB24,N},BitShift{ARGB32,N}}, val::Unsigned) = _map(UInt8, BS{N}, val)
map1{N}(mapi::Union{BitShift{RGB24,N},BitShift{ARGB32,N}}, val::UFixed) = _map(UFixed8, BS{N}, val)
map1{CT<:Colorant,N}(mapi::BitShift{CT,N}, val::UFixed) = _map(eltype(CT), BS{N}, val)

## Clamp types
# The Clamp types just enforce bounds, but do not scale or offset

# Types and constructors
abstract AbstractClamp{T} <: MapInfo{T}
"""
`ClampMin(T, minvalue)` is a `MapInfo` object that clamps pixel values
to be greater than or equal to `minvalue` before converting to type `T`.

See also: `ClampMax`, `ClampMinMax`.
"""
immutable ClampMin{T,From} <: AbstractClamp{T}
    min::From
end
ClampMin{T,From}(::Type{T}, min::From) = ClampMin{T,From}(min)
ClampMin{T}(min::T) = ClampMin{T,T}(min)
"""
`ClampMax(T, maxvalue)` is a `MapInfo` object that clamps pixel values
to be less than or equal to `maxvalue` before converting to type `T`.

See also: `ClampMin`, `ClampMinMax`.
"""
immutable ClampMax{T,From} <: AbstractClamp{T}
    max::From
end
ClampMax{T,From}(::Type{T}, max::From) = ClampMax{T,From}(max)
ClampMax{T}(max::T) = ClampMax{T,T}(max)
immutable ClampMinMax{T,From} <: AbstractClamp{T}
    min::From
    max::From
end
"""
`ClampMinMax(T, minvalue, maxvalue)` is a `MapInfo` object that clamps
pixel values to be between `minvalue` and `maxvalue` before converting
to type `T`.

See also: `ClampMin`, `ClampMax`, and `Clamp`.
"""
ClampMinMax{T,From}(::Type{T}, min::From, max::From) = ClampMinMax{T,From}(min,max)
ClampMinMax{T}(min::T, max::T) = ClampMinMax{T,T}(min,max)
"""
`Clamp(C)` is a `MapInfo` object that clamps color values to be within
gamut.  For example,

```
map(Clamp(RGB{U8}), RGB(1.2, -0.4, 0.6)) === RGB{U8}(1, 0, 0.6)
```
"""
immutable Clamp{T} <: AbstractClamp{T} end
Clamp{T}(::Type{T}) = Clamp{T}()

similar{T,F}(mapi::ClampMin, ::Type{T}, ::Type{F}) = ClampMin{T,F}(convert(F, mapi.min))
similar{T,F}(mapi::ClampMax, ::Type{T}, ::Type{F}) = ClampMax{T,F}(convert(F, mapi.max))
similar{T,F}(mapi::ClampMinMax, ::Type{T}, ::Type{F}) = ClampMin{T,F}(convert(F, mapi.min), convert(F, mapi.max))
similar{T,F}(mapi::Clamp, ::Type{T}, ::Type{F}) = Clamp{T}()

# Implementation
map{T<:Real,F<:Real}(mapi::ClampMin{T,F}, val::F) = convert(T, max(val, mapi.min))
map{T<:Real,F<:Real}(mapi::ClampMax{T,F}, val::F) = convert(T, min(val, mapi.max))
map{T<:Real,F<:Real}(mapi::ClampMinMax{T,F}, val::F) = convert(T,min(max(val, mapi.min), mapi.max))
map{T<:Fractional,F<:Real}(mapi::ClampMin{Gray{T},F}, val::F) = convert(Gray{T}, max(val, mapi.min))
map{T<:Fractional,F<:Real}(mapi::ClampMax{Gray{T},F}, val::F) = convert(Gray{T}, min(val, mapi.max))
map{T<:Fractional,F<:Real}(mapi::ClampMinMax{Gray{T},F}, val::F) = convert(Gray{T},min(max(val, mapi.min), mapi.max))
map{T<:Fractional,F<:Fractional}(mapi::ClampMin{Gray{T},F}, val::Gray{F}) = convert(Gray{T}, max(val, mapi.min))
map{T<:Fractional,F<:Fractional}(mapi::ClampMax{Gray{T},F}, val::Gray{F}) = convert(Gray{T}, min(val, mapi.max))
map{T<:Fractional,F<:Fractional}(mapi::ClampMinMax{Gray{T},F}, val::Gray{F}) = convert(Gray{T},min(max(val, mapi.min), mapi.max))
map{T<:Fractional,F<:Fractional}(mapi::ClampMin{Gray{T},Gray{F}}, val::Gray{F}) = convert(Gray{T}, max(val, mapi.min))
map{T<:Fractional,F<:Fractional}(mapi::ClampMax{Gray{T},Gray{F}}, val::Gray{F}) = convert(Gray{T}, min(val, mapi.max))
map{T<:Fractional,F<:Fractional}(mapi::ClampMinMax{Gray{T},Gray{F}}, val::Gray{F}) = convert(Gray{T},min(max(val, mapi.min), mapi.max))
map1{T<:Union{RGB24,ARGB32},F<:Fractional}(mapi::ClampMin{T,F}, val::F) = convert(UFixed8, max(val, mapi.min))
map1{T<:Union{RGB24,ARGB32},F<:Fractional}(mapi::ClampMax{T,F}, val::F) = convert(UFixed8, min(val, mapi.max))
map1{T<:Union{RGB24,ARGB32},F<:Fractional}(mapi::ClampMinMax{T,F}, val::F) = convert(UFixed8,min(max(val, mapi.min), mapi.max))
map1{CT<:Colorant,F<:Fractional}(mapi::ClampMin{CT,F}, val::F) = convert(eltype(CT), max(val, mapi.min))
map1{CT<:Colorant,F<:Fractional}(mapi::ClampMax{CT,F}, val::F) = convert(eltype(CT), min(val, mapi.max))
map1{CT<:Colorant,F<:Fractional}(mapi::ClampMinMax{CT,F}, val::F) = convert(eltype(CT), min(max(val, mapi.min), mapi.max))

map{To<:Real}(::Clamp{To}, val::Real) = clamp01(To, val)
map{To<:Real}(::Clamp{Gray{To}}, val::AbstractGray) = Gray(clamp01(To, val.val))
map{To<:Real}(::Clamp{Gray{To}}, val::Real) = Gray(clamp01(To, val))
map1{CT<:AbstractRGB}(::Clamp{CT}, val::Real) = clamp01(eltype(CT), val)
map1{P<:TransparentRGB}(::Clamp{P}, val::Real) = clamp01(eltype(P), val)

# Also available as a stand-alone function
clamp01{T}(::Type{T}, x::Real) = convert(T, min(max(x, zero(x)), one(x)))
clamp01(x::Real) = clamp01(typeof(x), x)
clamp01(x::Colorant) = clamp01(typeof(x), x)
clamp01{Cdest<:AbstractRGB   }(::Type{Cdest}, x::AbstractRGB)    = (To = eltype(Cdest);
    Cdest(clamp01(To, red(x)), clamp01(To, green(x)), clamp01(To, blue(x))))
clamp01{Pdest<:TransparentRGB}(::Type{Pdest}, x::TransparentRGB) = (To = eltype(Pdest);
    Pdest(clamp01(To, red(x)), clamp01(To, green(x)), clamp01(To, blue(x)), clamp01(To, alpha(x))))

# clamp is generic for any colorspace; this version does the right thing for any RGB type
clamp(x::Union{AbstractRGB, TransparentRGB}) = clamp01(x)

## ScaleMinMax
"""
`ScaleMinMax(T, min, max, [scalefactor])` is a `MapInfo` object that
clamps the image at the specified `min`/`max` values, subtracts the
`min` value, scales the result by multiplying by `scalefactor`, and
finally converts to type `T`.  If `scalefactor` is not specified, it
defaults to scaling the interval `[min,max]` to `[0,1]`.

Alternative constructors include `ScaleMinMax(T, img)` for which
`min`, `max`, and `scalefactor` are computed from the minimum and
maximum values found in `img`.

See also: `ScaleMinMaxNaN`, `ScaleAutoMinMax`, `MapNone`, `BitShift`.
"""
immutable ScaleMinMax{To,From,S<:AbstractFloat} <: MapInfo{To}
    min::From
    max::From
    s::S

    function ScaleMinMax(min, max, s)
        min >= max && error("min must be smaller than max")
        new(min, max, s)
    end
end

ScaleMinMax{To,From}(::Type{To}, min::From, max::From, s::AbstractFloat) = ScaleMinMax{To,From,typeof(s)}(min, max, s)
ScaleMinMax{To<:Union{Fractional,Colorant},From}(::Type{To}, mn::From, mx::From) = ScaleMinMax(To, mn, mx, 1.0f0/(convert(Float32, mx)-convert(Float32, mn)))

# ScaleMinMax constructors that take AbstractArray input
ScaleMinMax{To,From<:Real}(::Type{To}, img::AbstractArray{From}, mn::Real, mx::Real) = ScaleMinMax(To, convert(From,mn), convert(From,mx), 1.0f0/(convert(Float32, convert(From, mx))-convert(Float32,convert(From, mn))))
ScaleMinMax{To,From<:Real}(::Type{To}, img::AbstractArray{Gray{From}}, mn::Real, mx::Real) = ScaleMinMax(To, convert(From,mn), convert(From,mx), 1.0f0/(convert(Float32, convert(From,mx))-convert(Float32, convert(From,mn))))
ScaleMinMax{To,From<:Real,R<:Real}(::Type{To}, img::AbstractArray{From}, mn::Gray{R}, mx::Gray{R}) = ScaleMinMax(To, convert(From,mn.val), convert(From,mx.val), 1.0f0/(convert(Float32, convert(From,mx.val))-convert(Float32, convert(From,mn.val))))
ScaleMinMax{To,From<:Real,R<:Real}(::Type{To}, img::AbstractArray{Gray{From}}, mn::Gray{R}, mx::Gray{R}) = ScaleMinMax(To, convert(From,mn.val), convert(From,mx.val), 1.0f0/(convert(Float32, convert(From,mx.val))-convert(Float32, convert(From,mn.val))))
ScaleMinMax{To}(::Type{To}, img::AbstractArray) = ScaleMinMax(To, img, minfinite(img), maxfinite(img))
ScaleMinMax{To,CV<:AbstractRGB}(::Type{To}, img::AbstractArray{CV}) = (imgr = reinterpret(eltype(CV), img); ScaleMinMax(To, minfinite(imgr), maxfinite(imgr)))

similar{T,F,To,From,S}(mapi::ScaleMinMax{To,From,S}, ::Type{T}, ::Type{F}) = ScaleMinMax{T,F,S}(convert(F,mapi.min), convert(F.mapi.max), mapi.s)

# Implementation
function map{To<:Union{Real,AbstractGray},From<:Union{Real,AbstractGray}}(mapi::ScaleMinMax{To,From}, val::From)
    g = gray(val)
    t = ifelse(g < mapi.min, zero(From), ifelse(g > mapi.max, mapi.max-mapi.min, g-mapi.min))
    convert(To, mapi.s*t)
end
function map{To<:Union{Real,AbstractGray},From<:Union{Real,AbstractGray}}(mapi::ScaleMinMax{To,From}, val::Union{Real,Colorant})
    map(mapi, convert(From, val))
end
function map1{To<:Union{RGB24,ARGB32},From<:Real}(mapi::ScaleMinMax{To,From}, val::From)
    t = ifelse(val < mapi.min, zero(From), ifelse(val > mapi.max, mapi.max-mapi.min, val-mapi.min))
    convert(UFixed8, mapi.s*t)
end
function map1{To<:Colorant,From<:Real}(mapi::ScaleMinMax{To,From}, val::From)
    t = ifelse(val < mapi.min, zero(From), ifelse(val > mapi.max, mapi.max-mapi.min, val-mapi.min))
    convert(eltype(To), mapi.s*t)
end
function map1{To<:Union{RGB24,ARGB32},From<:Real}(mapi::ScaleMinMax{To,From}, val::Union{Real,Colorant})
    map1(mapi, convert(From, val))
end
function map1{To<:Colorant,From<:Real}(mapi::ScaleMinMax{To,From}, val::Union{Real,Colorant})
    map1(mapi, convert(From, val))
end

## ScaleSigned
"""
`ScaleSigned(T, scalefactor)` is a `MapInfo` object designed for
visualization of images where the pixel's sign has special meaning.
It multiplies the pixel value by `scalefactor`, then clamps to the
interval `[-1,1]`. If `T` is a floating-point type, it stays in this
representation.  If `T` is an `AbstractRGB`, then it is encoded as a
magenta (positive)/green (negative) image, with the intensity of the
color proportional to the clamped absolute value.
"""
immutable ScaleSigned{T, S<:AbstractFloat} <: MapInfo{T}
    s::S
end
ScaleSigned{T}(::Type{T}, s::AbstractFloat) = ScaleSigned{T, typeof(s)}(s)

ScaleSigned{T}(::Type{T}, img::AbstractArray) = ScaleSigned(T, 1.0f0/maxabsfinite(img))
ScaleSigned(img::AbstractArray) = ScaleSigned(Float32, img)

similar{T,To,S}(mapi::ScaleSigned{To,S}, ::Type{T}, ::Type) = ScaleSigned{T,S}(mapi.s)

map{T}(mapi::ScaleSigned{T}, val::Real) = convert(T, clamppm(mapi.s*val))
function map{C<:AbstractRGB}(mapi::ScaleSigned{C}, val::Real)
    x = clamppm(mapi.s*val)
    g = UFixed8(abs(x))
    ifelse(x >= 0, C(g, zero(UFixed8), g), C(zero(UFixed8), g, zero(UFixed8)))
end

clamppm(x::Real) = ifelse(x >= 0, min(x, one(x)), max(x, -one(x)))

## ScaleAutoMinMax
# Works only on whole arrays, not values
"""
`ScaleAutoMinMax(T)` constructs a `MapInfo` object that causes images
to be dynamically scaled to their specific min/max values, using the
same algorithm for `ScaleMinMax`. When displaying a movie, the min/max
will be recalculated for each frame, so this can result in
inconsistent contrast scaling.
"""
immutable ScaleAutoMinMax{T} <: MapInfo{T} end
ScaleAutoMinMax{T}(::Type{T}) = ScaleAutoMinMax{T}()
ScaleAutoMinMax() = ScaleAutoMinMax{UFixed8}()

similar{T}(mapi::ScaleAutoMinMax, ::Type{T}, ::Type) = ScaleAutoMinMax{T}()

## NaN-nulling mapping
"""
`ScaleMinMaxNaN(smm)` constructs a `MapInfo` object from a
`ScaleMinMax` object `smm`, with the additional property that `NaN`
values map to zero.

See also: `ScaleMinMax`.
"""
immutable ScaleMinMaxNaN{To,From,S} <: MapInfo{To}
    smm::ScaleMinMax{To,From,S}
end

"""
`Clamp01NaN(T)` or `Clamp01NaN(img)` constructs a `MapInfo` object
that clamps grayscale or color pixels to the interval `[0,1]`, sending
`NaN` pixels to zero.
"""
immutable Clamp01NaN{T} <: MapInfo{T} end

Clamp01NaN{T}(A::AbstractArray{T}) = Clamp01NaN{T}()

# Implementation
similar{T,F,To,From,S}(mapi::ScaleMinMaxNaN{To,From,S}, ::Type{T}, ::Type{F}) = ScaleMinMaxNaN{T,F,S}(similar(mapi.smm, T, F))
similar{T}(mapi::Clamp01NaN, ::Type{T}, ::Type) = Clamp01NaN{T}()

Base.map{To}(smmn::ScaleMinMaxNaN{To}, g::Number) = isnan(g) ? zero(To) : map(smmn.smm, g)
Base.map{To}(smmn::ScaleMinMaxNaN{To}, g::Gray) = isnan(g) ? zero(To) : map(smmn.smm, g)

function Base.map{T<:RGB}(::Clamp01NaN{T}, c::AbstractRGB)
    r, g, b = red(c), green(c), blue(c)
    if isnan(r) || isnan(g) || isnan(b)
        return T(0,0,0)
    end
    T(clamp(r, 0, 1), clamp(g, 0, 1), clamp(b, 0, 1))
end
function Base.map{T<:Union{Fractional,Gray}}(::Clamp01NaN{T}, c::Union{Fractional,AbstractGray})
    g = gray(c)
    if isnan(g)
        return T(0)
    end
    T(clamp(g, 0, 1))
end

# Conversions to RGB{T}, RGBA{T}, RGB24, ARGB32,
# for grayscale, AbstractRGB, and abstract ARGB inputs.
# This essentially "vectorizes" map over a single pixel's color channels using map1
for SI in (MapInfo, AbstractClamp)
    for ST in subtypes(SI)
        ST.abstract && continue
        ST == ScaleSigned && continue  # ScaleSigned gives an RGB from a scalar, so don't "vectorize" it
        @eval begin
            # Grayscale and GrayAlpha inputs
            map(mapi::$ST{RGB24}, g::Gray) = map(mapi, g.val)
            map(mapi::$ST{RGB24}, g::Real) = (x = map1(mapi, g); convert(RGB24, RGB{UFixed8}(x,x,x)))
            function map(mapi::$ST{RGB24}, g::AbstractFloat)
                if isfinite(g)
                    x = map1(mapi, g)
                    convert(RGB24, RGB{UFixed8}(x,x,x))
                else
                    RGB24(0)
                end
            end
            map{G<:Gray}(mapi::$ST{RGB24}, g::TransparentColor{G}) = map(mapi, gray(g))
            map(mapi::$ST{ARGB32}, g::Gray) = map(mapi, g.val)
            function map(mapi::$ST{ARGB32}, g::Real)
                x = map1(mapi, g)
                convert(ARGB32, ARGB{UFixed8}(x,x,x,0xffuf8))
            end
            function map{G<:Gray}(mapi::$ST{ARGB32}, g::TransparentColor{G})
                x = map1(mapi, gray(g))
                convert(ARGB32, ARGB{UFixed8}(x,x,x,map1(mapi, g.alpha)))
            end
        end
        for O in (:RGB, :BGR)
            @eval begin
                map{T}(mapi::$ST{$O{T}}, g::Gray) = map(mapi, g.val)
                function map{T}(mapi::$ST{$O{T}}, g::Real)
                    x = map1(mapi, g)
                    $O{T}(x,x,x)
                end
            end
        end
        for OA in (:RGBA, :ARGB, :BGRA)
            exAlphaGray = ST == MapNone ? :nothing : quote
                function map{T,G<:Gray}(mapi::$ST{$OA{T}}, g::TransparentColor{G})
                    x = map1(mapi, gray(g))
                    $OA{T}(x,x,x,map1(mapi, g.alpha))
                end  # avoids an ambiguity warning with MapNone definitions
            end
            @eval begin
                map{T}(mapi::$ST{$OA{T}}, g::Gray) = map(mapi, g.val)
                function map{T}(mapi::$ST{$OA{T}}, g::Real)
                    x = map1(mapi, g)
                    $OA{T}(x,x,x)
                end
                $exAlphaGray
            end
        end
        @eval begin
            # AbstractRGB and abstract ARGB inputs
            map(mapi::$ST{RGB24}, rgb::AbstractRGB) =
                convert(RGB24, RGB{UFixed8}(map1(mapi, red(rgb)), map1(mapi, green(rgb)), map1(mapi, blue(rgb))))
            map{C<:AbstractRGB, TC}(mapi::$ST{RGB24}, argb::TransparentColor{C,TC}) =
                convert(RGB24, RGB{UFixed8}(map1(mapi, red(argb)), map1(mapi, green(argb)),
                                            map1(mapi, blue(argb))))
            map{C<:AbstractRGB, TC}(mapi::$ST{ARGB32}, argb::TransparentColor{C,TC}) =
                convert(ARGB32, ARGB{UFixed8}(map1(mapi, red(argb)), map1(mapi, green(argb)),
                                              map1(mapi, blue(argb)), map1(mapi, alpha(argb))))
            map(mapi::$ST{ARGB32}, rgb::AbstractRGB) =
                convert(ARGB32, ARGB{UFixed8}(map1(mapi, red(rgb)), map1(mapi, green(rgb)), map1(mapi, blue(rgb))))
        end
        for O in (:RGB, :BGR)
            @eval begin
                map{T}(mapi::$ST{$O{T}}, rgb::AbstractRGB) =
                    $O{T}(map1(mapi, red(rgb)), map1(mapi, green(rgb)), map1(mapi, blue(rgb)))
                map{T,C<:AbstractRGB, TC}(mapi::$ST{$O{T}}, argb::TransparentColor{C,TC}) =
                    $O{T}(map1(mapi, red(argb)), map1(mapi, green(argb)), map1(mapi, blue(argb)))
            end
        end
        for OA in (:RGBA, :ARGB, :BGRA)
            @eval begin
                map{T, C<:AbstractRGB, TC}(mapi::$ST{$OA{T}}, argb::TransparentColor{C,TC}) =
                    $OA{T}(map1(mapi, red(argb)), map1(mapi, green(argb)),
                            map1(mapi, blue(argb)), map1(mapi, alpha(argb)))
                map{T}(mapi::$ST{$OA{T}}, argb::ARGB32) = map(mapi, convert(RGBA{UFixed8}, argb))
                map{T}(mapi::$ST{$OA{T}}, rgb::AbstractRGB) =
                    $OA{T}(map1(mapi, red(rgb)), map1(mapi, green(rgb)), map1(mapi, blue(rgb)))
                map{T}(mapi::$ST{$OA{T}}, rgb::RGB24) = map(mapi, convert(RGB{UFixed8}, argb))
            end
        end
    end
end

# # Apply to any Colorant
# map(f::Callable, x::Color) = f(x)
# map(mapi, x::Color) = map(mapi, convert(RGB, x))
# map{C<:Color, TC}(f::Callable, x::TransparentColor{C, TC}) = f(convert(ARGB, x))
# map{C<:Color, TC}(mapi, x::TransparentColor{C, TC}) = map(mapi, convert(ARGB, x))

## Fallback definitions of map() for array types

function map{T}(mapi::MapInfo{T}, img::AbstractArray)
    out = similar(img, T)
    map!(mapi, out, img)
end

map{C<:Colorant,R<:Real}(mapi::MapNone{C}, img::AbstractImageDirect{R}) = mapcd(mapi, img)  # ambiguity resolution
map{C<:Colorant,R<:Real}(mapi::MapInfo{C}, img::AbstractImageDirect{R}) = mapcd(mapi, img)
function mapcd{C<:Colorant,R<:Real}(mapi::MapInfo{C}, img::AbstractImageDirect{R})
    # For this case we have to check whether color is defined along an array axis
    cd = colordim(img)
    if cd > 0
        dims = setdiff(1:ndims(img), cd)
        out = similar(img, C, size(img)[dims])
        map!(mapi, out, img, TypeConst{cd})
    else
        out = similar(img, C)
        map!(mapi, out, img)
    end
    out   # note this isn't type-stable
end

function map{T<:Colorant}(mapi::MapInfo{T}, img::AbstractImageIndexed)
    out = Image(Array(T, size(img)), properties(img))
    map!(mapi, out, img)
end

map!{T,T1,T2,N}(mapi::MapInfo{T1}, out::AbstractArray{T,N}, img::AbstractArray{T2,N}) =
    _map_a!(mapi, out, img)
function _map_a!{T,T1,T2,N}(mapi::MapInfo{T1}, out::AbstractArray{T,N}, img::AbstractArray{T2,N})
    mi = take(mapi, img)
    dimg = data(img)
    dout = data(out)
    size(dout) == size(dimg) || throw(DimensionMismatch())
    for I in eachindex(dout, dimg)
        @inbounds dout[I] = map(mi, dimg[I])
    end
    out
end

take(mapi::MapInfo, img::AbstractArray) = mapi
take{T}(mapi::ScaleAutoMinMax{T}, img::AbstractArray) = ScaleMinMax(T, img)

# Indexed images (colormaps)
map!{T,T1,N}(mapi::MapInfo{T}, out::AbstractArray{T,N}, img::AbstractImageIndexed{T1,N}) =
    _mapindx!(mapi, out, img)
function _mapindx!{T,T1,N}(mapi::MapInfo{T}, out::AbstractArray{T,N}, img::AbstractImageIndexed{T1,N})
    dimg = data(img)
    dout = data(out)
    cmap = map(mapi, img.cmap)
    for I in eachindex(dout, dimg)
        @inbounds dout[I] = cmap[dimg[I]]
    end
    out
end

# For when color is encoded along dimension CD
# NC is the number of color channels
# This is a very flexible implementation: color can be stored along any dimension, and it handles conversions to
# many different colorspace representations.
for (CT, NC) in ((Union{AbstractRGB,RGB24}, 3), (Union{RGBA,ARGB,ARGB32}, 4), (Union{AGray,GrayA,AGray32}, 2))
    for N = 1:4
        N1 = N+1
        @eval begin
function map!{T<:$CT,T1,T2,CD}(mapi::MapInfo{T}, out::AbstractArray{T1,$N}, img::AbstractArray{T2,$N1}, ::Type{TypeConst{CD}})
    mi = take(mapi, img)
    dimg = data(img)
    dout = data(out)
    # Set up the index along the color axis
    # We really only need dimension CD, but this will suffice
    @nexprs $NC k->(@nexprs $N1 d->(j_k_d = k))
    # Loop over all the elements in the output, performing the conversion on each color component
    @nloops $N i dout d->(d<CD ? (@nexprs $NC k->(j_k_d = i_d)) : (@nexprs $NC k->(j_k_{d+1} = i_d))) begin
        @inbounds @nref($N, dout, i) = @ncall $NC T k->(map1(mi, @nref($N1, dimg, j_k)))
    end
    out
end
        end
    end
end

#### MapInfo defaults
# Each "client" can define its own methods. "clients" include UFixed,
# RGB24/ARGB32, and ImageMagick

const bitshiftto8 = ((UFixed10, 2), (UFixed12, 4), (UFixed14, 6), (UFixed16, 8))

# typealias GrayType{T<:Fractional} Union{T, Gray{T}}
typealias GrayArray{T<:Fractional} Union{AbstractArray{T}, AbstractArray{Gray{T}}}
# note, though, that we need to override for AbstractImage in case the
# "colorspace" property is defined differently

# mapinfo{T<:Union{Real,Colorant}}(::Type{T}, img::AbstractArray{T}) = MapNone(img)
"""
`mapi = mapinf(T, img)` returns a `MapInfo` object that is deemed
appropriate for converting pixels of `img` to be of type `T`. `T` can
either be a specific type (e.g., `RGB24`), or you can specify an
abstract type like `Clamp` and it will return one of the `Clamp`
family of `MapInfo` objects.

You can define your own rules for `mapinfo`.  For example, the
`ImageMagick` package defines methods for how pixels values should be
converted before saving images to disk.
"""
mapinfo{T<:UFixed}(::Type{T}, img::AbstractArray{T}) = MapNone(img)
mapinfo{T<:AbstractFloat}(::Type{T}, img::AbstractArray{T}) = MapNone(img)

# Grayscale methods
mapinfo(::Type{UFixed8}, img::GrayArray{UFixed8}) = MapNone{UFixed8}()
mapinfo(::Type{Gray{UFixed8}}, img::GrayArray{UFixed8}) = MapNone{Gray{UFixed8}}()
mapinfo(::Type{GrayA{UFixed8}}, img::AbstractArray{GrayA{UFixed8}}) = MapNone{GrayA{UFixed8}}()
for (T,n) in bitshiftto8
    @eval mapinfo(::Type{UFixed8}, img::GrayArray{$T}) = BitShift{UFixed8,$n}()
    @eval mapinfo(::Type{Gray{UFixed8}}, img::GrayArray{$T}) = BitShift{Gray{UFixed8},$n}()
    @eval mapinfo(::Type{GrayA{UFixed8}}, img::AbstractArray{GrayA{$T}}) = BitShift{GrayA{UFixed8},$n}()
end
mapinfo{T<:UFixed,F<:AbstractFloat}(::Type{T}, img::GrayArray{F}) = ClampMinMax(T, zero(F), one(F))
mapinfo{T<:UFixed,F<:AbstractFloat}(::Type{Gray{T}}, img::GrayArray{F}) = ClampMinMax(Gray{T}, zero(F), one(F))
mapinfo{T<:AbstractFloat, R<:Real}(::Type{T}, img::AbstractArray{R}) = MapNone(T)

mapinfo(::Type{RGB24}, img::Union{AbstractArray{Bool}, BitArray}) = MapNone{RGB24}()
mapinfo(::Type{ARGB32}, img::Union{AbstractArray{Bool}, BitArray}) = MapNone{ARGB32}()
mapinfo{F<:Fractional}(::Type{RGB24}, img::GrayArray{F}) = ClampMinMax(RGB24, zero(F), one(F))
mapinfo{F<:Fractional}(::Type{ARGB32}, img::AbstractArray{F}) = ClampMinMax(ARGB32, zero(F), one(F))

# Color->Color methods
mapinfo(::Type{RGB{UFixed8}}, img) = MapNone{RGB{UFixed8}}()
mapinfo(::Type{RGBA{UFixed8}}, img) = MapNone{RGBA{UFixed8}}()
for (T,n) in bitshiftto8
    @eval mapinfo(::Type{RGB{UFixed8}}, img::AbstractArray{RGB{$T}}) = BitShift{RGB{UFixed8},$n}()
    @eval mapinfo(::Type{RGBA{UFixed8}}, img::AbstractArray{RGBA{$T}}) = BitShift{RGBA{UFixed8},$n}()
end
mapinfo{F<:Fractional}(::Type{RGB{UFixed8}}, img::AbstractArray{RGB{F}}) = Clamp(RGB{UFixed8})
mapinfo{F<:Fractional}(::Type{RGBA{UFixed8}}, img::AbstractArray{RGBA{F}}) = Clamp(RGBA{UFixed8})

# Color->RGB24/ARGB32
mapinfo(::Type{RGB24}, img::AbstractArray{RGB24}) = MapNone{RGB24}()
mapinfo(::Type{ARGB32}, img::AbstractArray{ARGB32}) = MapNone{ARGB32}()
for C in tuple(subtypes(AbstractRGB)..., Gray)
    C == RGB24 && continue
    @eval mapinfo(::Type{RGB24}, img::AbstractArray{$C{UFixed8}}) = MapNone{RGB24}()
    @eval mapinfo(::Type{ARGB32}, img::AbstractArray{$C{UFixed8}}) = MapNone{ARGB32}()
    for (T, n) in bitshiftto8
        @eval mapinfo(::Type{RGB24}, img::AbstractArray{$C{$T}}) = BitShift{RGB24, $n}()
        @eval mapinfo(::Type{ARGB32}, img::AbstractArray{$C{$T}}) = BitShift{ARGB32, $n}()
    end
    @eval mapinfo{F<:AbstractFloat}(::Type{RGB24}, img::AbstractArray{$C{F}}) = ClampMinMax(RGB24, zero(F), one(F))
    @eval mapinfo{F<:AbstractFloat}(::Type{ARGB32}, img::AbstractArray{$C{F}}) = ClampMinMax(ARGB32, zero(F), one(F))
    for AC in subtypes(TransparentColor)
        length(AC.parameters) == 2 || continue
        @eval mapinfo(::Type{ARGB32}, img::AbstractArray{$AC{$C{UFixed8},UFixed8}}) = MapNone{ARGB32}()
        @eval mapinfo(::Type{RGB24}, img::AbstractArray{$AC{$C{UFixed8},UFixed8}}) = MapNone{RGB24}()
        for (T, n) in bitshiftto8
            @eval mapinfo(::Type{ARGB32}, img::AbstractArray{$AC{$C{$T},$T}}) = BitShift{ARGB32, $n}()
            @eval mapinfo(::Type{RGB24}, img::AbstractArray{$AC{$C{$T},$T}}) = BitShift{RGB24, $n}()
        end
        @eval mapinfo{F<:AbstractFloat}(::Type{ARGB32}, img::AbstractArray{$AC{$C{F},F}}) = ClampMinMax(ARGB32, zero(F), one(F))
        @eval mapinfo{F<:AbstractFloat}(::Type{RGB24}, img::AbstractArray{$AC{$C{F},F}}) = ClampMinMax(RGB24, zero(F), one(F))
    end
end

mapinfo{CT<:Colorant}(::Type{RGB24},  img::AbstractArray{CT}) = MapNone{RGB24}()
mapinfo{CT<:Colorant}(::Type{ARGB32}, img::AbstractArray{CT}) = MapNone{ARGB32}()

# UInt32 conversions will use ARGB32 for images that have an alpha channel,
# and RGB24 when not
mapinfo{CV<:Union{Fractional,Color,AbstractGray}}(::Type{UInt32}, img::AbstractArray{CV}) = mapinfo(RGB24, img)
mapinfo{CV<:TransparentColor}(::Type{UInt32}, img::AbstractArray{CV}) = mapinfo(ARGB32, img)
mapinfo(::Type{UInt32}, img::Union{AbstractArray{Bool},BitArray}) = mapinfo(RGB24, img)
mapinfo(::Type{UInt32}, img::AbstractArray{UInt32}) = MapNone{UInt32}()

# Clamping mapinfo client. Converts to RGB and uses UFixed, clamping
# floating-point values to [0,1].
mapinfo{T<:UFixed}(::Type{Clamp}, img::AbstractArray{T}) = MapNone{T}()
mapinfo{T<:AbstractFloat}(::Type{Clamp}, img::AbstractArray{T}) = ClampMinMax(UFixed8, zero(T), one(T))
let handled = Set()
for ACV in (Color, AbstractRGB)
    for CV in subtypes(ACV)
        (length(CV.parameters) == 1 && !(CV.abstract)) || continue
        CVnew = CV<:AbstractGray ? Gray : RGB
        @eval mapinfo{T<:UFixed}(::Type{Clamp}, img::AbstractArray{$CV{T}}) = MapNone{$CVnew{T}}()
        @eval mapinfo{CV<:$CV}(::Type{Clamp}, img::AbstractArray{CV}) = Clamp{$CVnew{UFixed8}}()
        CVnew = CV<:AbstractGray ? Gray : BGR
        AC, CA       = alphacolor(CV), coloralpha(CV)
        if AC in handled
            continue
        end
        push!(handled, AC)
        ACnew, CAnew = alphacolor(CVnew), coloralpha(CVnew)
        @eval begin
            mapinfo{T<:UFixed}(::Type{Clamp}, img::AbstractArray{$AC{T}}) = MapNone{$ACnew{T}}()
            mapinfo{P<:$AC}(::Type{Clamp}, img::AbstractArray{P}) = Clamp{$ACnew{UFixed8}}()
            mapinfo{T<:UFixed}(::Type{Clamp}, img::AbstractArray{$CA{T}}) = MapNone{$CAnew{T}}()
            mapinfo{P<:$CA}(::Type{Clamp}, img::AbstractArray{P}) = Clamp{$CAnew{UFixed8}}()
        end
    end
end
end
mapinfo(::Type{Clamp}, img::AbstractArray{RGB24}) = MapNone{RGB{UFixed8}}()
mapinfo(::Type{Clamp}, img::AbstractArray{ARGB32}) = MapNone{BGRA{UFixed8}}()

# Backwards-compatibility
uint32color(img) = map(mapinfo(UInt32, img), img)
uint32color!(buf, img::AbstractArray) = map!(mapinfo(UInt32, img), buf, img)
uint32color!(buf, img::AbstractArray, mi::MapInfo) = map!(mi, buf, img)
uint32color!{T,N}(buf::Array{UInt32,N}, img::AbstractImageDirect{T,N}) =
    map!(mapinfo(UInt32, img), buf, img)
uint32color!{T,N,N1}(buf::Array{UInt32,N}, img::AbstractImageDirect{T,N1}) =
    map!(mapinfo(UInt32, img), buf, img, TypeConst{colordim(img)})
uint32color!{T,N}(buf::Array{UInt32,N}, img::AbstractImageDirect{T,N}, mi::MapInfo) =
    map!(mi, buf, img)
uint32color!{T,N,N1}(buf::Array{UInt32,N}, img::AbstractImageDirect{T,N1}, mi::MapInfo) =
    map!(mi, buf, img, TypeConst{colordim(img)})

"""
```
imgsc = sc(img)
imgsc = sc(img, min, max)
```

Applies default or specified `ScaleMinMax` mapping to the image.
"""
sc(img::AbstractArray) = map(ScaleMinMax(UFixed8, img), img)
sc(img::AbstractArray, mn::Real, mx::Real) = map(ScaleMinMax(UFixed8, img, mn, mx), img)

for (fn,T) in ((:float32, Float32), (:float64, Float64), (:ufixed8, UFixed8),
               (:ufixed10, UFixed10), (:ufixed12, UFixed12), (:ufixed14, UFixed14),
               (:ufixed16, UFixed16))
    @eval begin
        function $fn{C<:Colorant}(A::AbstractArray{C})
            newC = eval(C.name.name){$T}
            convert(Array{newC}, A)
        end
        $fn{C<:Colorant}(img::AbstractImage{C}) = shareproperties(img, $fn(data(img)))
    end
end

ufixedsc{T<:UFixed}(::Type{T}, img::AbstractImageDirect) = map(mapinfo(T, img), img)
ufixed8sc(img::AbstractImageDirect) = ufixedsc(UFixed8, img)