CrossClj

0.6.4-alpha1 docs

SourceDocs



RECENT
    VARS
    abort-fn
    check-job-status
    combine
    combiner-config
    config
    defremotes
    ensure-output-paths!
    error
    execute
    fexecute
    flatten-graph
    flatten-graph*
    gen-id
    graph-delay
    graph-future
    input
    job-complete?
    job-name
    job-ran?
    job-running?
    job-successful?
    map
    map-only
    mapper-config
    node-fn
    node-id
    node-job
    output
    output*
    partition
    partition*
    partitioner-config
    re-input
    reduce
    reducer-config
    remote-config
    run-graph
    run-graph*
    run-job
    shuffle
    shuffle*
    shuffle-classes?
    sink
    source
    stage

    « Index of all namespaces of this project

    Private
    (abort-fn job)
    Function for terminating job, killing it and cleaning up output path(s).
    
    Private
    (check-job-status f job)
    (combine node cls)(combine node var & args)
    Add combine task to job node node, as implemented by Reducer class cls,
    or Clojure var var and optional args.
    (config node & steps)
    Add arbitrary configuration steps to node, which may be either a single job
    node or a vector of job nodes.
    macroPrivate
    (defremotes & args)
    Private
    (ensure-output-paths! job)
    macroPrivate
    (error verb node)
    (execute graph conf jname)
    Execute Hadoop jobs for the job graph graph, which should be a job graph
    leaf node or vector of leaf nodes.  Jobs are configured starting with base
    configuration conf and named based on the string jname.  Returns a vector of
    the distributed sequences produced by the job graph leaves.
    (fexecute graph conf jname)
    As per execute, but require and return only a single result desq;
    almost `(comp first execute)`.
    Private
    (flatten-graph graph)
    Flatten job graph graph into a vector of job nodes annotated with vector
    positional job ID as :jid and by-ID dependencies as :requires.  Return tuple
    of the nodes-vector and a vector of the leaf-node job-IDs.
    Private
    (flatten-graph* graph)
    Private
    (gen-id)
    Return application-unique input/output ID.
    
    Private
    (graph-delay f inputs)
    Delay result of applying function f to the values held by delays inputs.
    
    Private
    (graph-future f inputs)
    Future result of applying function f to the values held by futures
    inputs.  Attempts to cancel all inputs upon any failures.
    (input node)
    Return a fresh :input-stage job graph node consuming from the provided dseq
    node.  If instead provided an :input-stage node or vector of such nodes,
    acts as the identity function.
    Private
    (job-name base n i)
    Job name for ith job of n produced from var-name base.
    
    Private
    (job-ran? job)
    True iff job ran or is currently running.
    
    Private
    (job-running? job)
    True iff job is currently running.
    
    multimethod
    (map node cls)(map node var & args)
    Add map task to job node node, as implemented by Mapper class cls, or
    Clojure var var and optional args.
    Private
    (map-only job)
    Configuration step for map-only jobs.
    
    multimethod
    (node-fn node conf jname)
    Return a function for executing the job defined by the job node node, using
    base configuration conf and job name jname.
    Private
    (node-id node)
    Application-unique node-identifier of node node.
    
    (node-job node conf jname)
    Hadoop Job for job node node, starting with base configuration conf
    and named jname.
    (output node dsink)(output node & named-dsinks)
    Add output task to job node node for writing to dsink or named-output
    name-dsink pairs named-dsinks.  Yields either a new :input-stage node
    reading from the written output or a vector of such nodes.
    Private
    (output* node dsink)
    Chain output task following job node node.
    
    (partition node step)(partition node step cls)(partition node step var & args)
    Add partition task to the provided job node node, as configured by step
    and optionally implemented by either Partitioner class cls or Clojure var
    var & optional args.  The node may be either a single job node or a vector
    of job nodes to co-group.  The step may be either a configuration step or a
    vector of the two map-output key & value classes.
    multimethodPrivate
    (partition* node step)(partition* node step cls)(partition* node step var & args)
    Internal dispatch multimethod for partition implementations.
    
    Private
    (re-input node dsink)
    (reduce node cls)(reduce node var & args)
    Add reduce task to job node node, as implemented by Reducer class cls,
    or Clojure var var and optional args.
    multimethodPrivate
    (remote-config alloc set-class cls)(remote-config alloc set-class uvar & args)
    Return config step for allocating remote task classes.
    
    (run-graph runner graph outputs)
    Execute the data-flow graph described by graph using the function executor
    runner.  Each key in graph identifies a particular entry.  Each value is a
    tuple of `(inputs, f)`, where inputs is a sequence of other graph keys and
    f is a function calculating that entry's result given inputs.  Returns a
    vector of the result entries for the keys in the collection outputs.
    Private
    (run-graph* runner graph results output)
    Build execution graph for the data-flow described by the map graph.
    The results map holds calculated result reference, and output is the key of
    the desired result.  Returns a tuple of updated `(results, result)`.
    (run-job job)
    Run job and wait synchronously for it to complete.  Kills the job on
    exceptions or JVM shutdown.  Unlike the Job#waitForCompletion() method, does
    not swallow InterruptedException.
    (shuffle [ckey] [[ckey cval]])
    Base shuffle configuration; sets map output key & value types to
    the classes ckey and cval respectively.
    Private
    (shuffle* ckey)(shuffle* ckey cval)
    Internal implementation of shuffle.
    
    Private
    (shuffle-classes? classes)
    True iff classes is a vector of two classes.
    
    Deprecated
    (sink node dsink)(sink node & named-dsinks)
    Deprecated alias for output.
    
    Deprecated
    (source dseq)
    Deprecated alias for input.
    
    Private
    (stage node & args)
    The job stage of job node node.