CrossClj

0.3.0 docs

SourceDocs



RECENT

    almonds

    Clojars

    Jan 21, 2015


    OWNER
    Murtaza Husain
    Pune
    murtaza52@gmail.com

    Readme

    Index of all namespaces


    « Project + dependencies

    A library for infrastructure automation

    The README below is fetched from the published project artifact. Some relative links may be broken.
    - [what](#sec-1)
    - [why](#sec-2)
    - [status](#sec-3)
    - [usage](#sec-4)
      - [Releases](#sec-4-1)
      - [Credentials](#sec-4-2)
    - [how](#sec-5)
      - [defining resources](#sec-5-1)
      - [state management](#sec-5-2)
      - [staging resources](#sec-5-3)
      - [diff](#sec-5-4)
      - [api functions](#sec-5-5)
      - [sync-resources](#sec-5-6)
      - [modifying resources](#sec-5-7)
      - [recreating resources](#sec-5-8)
    - [acknowledgements](#sec-6)
    

    what

    almonds is a library for realizing the ideal of infrastructure as code. It operates in the same space that of AWS’s Cloud Formation and Hashicorp’s Terraform. It takes inspiration from both, tries to address their shortcomings, and introduces features which are developer friendly.

    why

    There are few problems associated with the current crop of tools -

    • State Management
      • Terraform: this is the Achille’s heel of terraform. It maintains state in a file which is its source of truth. This file has to be shared/synced when multiple developers are working simultaneously, and is a pain. The scenario (though unlikely) of the state file being irrecoverable will be disastrous.
      • CloudFormation: cloudformation does a much better job, it maintains identity of each resource, and recreates the state every time it is run. However there are horror stories of its state getting corrupted during failed updates, and the only way out is to call in the AWS support team.
      • Almonds: almonds borrows cloudformation’s way of specifying the identity of resources. However this functionality is explicit and user can choose to set the identity. The state is also always recreated and is explicit to the user, who can inspect both the local state and the remote state. The user can also diff between the local and remote state’s (an idea borrowed from terraform), and even compare individual resources between state’s.
    • Unpredictability
      • Terraform and CloudFormation: both try to be /“intelligent”/ by calculating the dependencies of resources, and then performing CRUD on them too. This leads to unwanted behaviour and nasty surprises. Example - If the security group of an instance is changed, then both the tools will also delete and recreate the instance. A simple update can turn into a nightmare. You never know which operation will succeed and which will fail.
      • Almonds: almonds is /“dumb”/, it leaves the intelligence to the user. It only performs the specific operation on the specific group of resources that are specified. It will not perform those operations on the dependencies by default, but this can be specified too. All operations are explicit and there are no surprises.
    • DSL
      • Terraform and CloudFormation: both of them provide an external DSL to specify resources. CloudFormation has a minimal one while terraform provides an extensive one. IMHO this is a major mistake, Terraform’s DSL is a pain to use, and looking at the DSL it seems they are trying to recreate a programming language, albiet a very clunky one.
      • Almonds: almonds will not provide an external DSL. Currently it can be used in you JVM projects as a library. It will provide limited command line functionality where the input will be a plain json file. Your favourite programming language can be used to generate the json if needed.
    • Coarse Grained
      • Terraform and CloudFormation: both of these are hammers, and your every operation better be a nail. Your context is a file/folder, and all crud operations will be applied on all resources in that context.
      • Almonds: it provides a very fine grained mechanism to group resources, and allows control over what operations should be run and in what order. Its a library, which provides a set of functions to deal with resources. These functions can now be composed to form new abstractions. This is perhaps the most important differentiator, almonds is a library.

    status

    almonds is a very young tool and you will encounter bugs. It currently only provides for the CRUD of a few EC2 resources, but has plans to support all EC2 resources in next few months. It can be also extended to include resources from providers other than EC2, if there is sufficient interest. PR’s / suggestions / criticisms are all very much welcome :)

    It can be used as a library in your clojure or JVM project. In near future you would also be able to run it from the command line with json as input. However the json will be plain vanilla json and no DSL will be added. The json can be generated using your preferred language.

    usage

    Releases

    [almonds "0.2.3"]
    
    • To use it as a maven dependency -
    <dependency>
      <groupId>almonds</groupId>
      <artifactId>almonds</artifactId>
      <version>0.2.3</version>
    </dependency>
    

    Credentials

    In your code set the aws credentials -

    (require [almonds.core :refer [set-aws-credentials]])
    
    (set-aws-credentials "aws-access-key" "aws-secret" "https://ec2.amazonaws.com")
    

    how

    defining resources

    • Resources are defined as a hash map.
    • Each resource has two mandatory properties -
      • :almonds-type: this key denotes the type of the resource and has to be from one of the types defined by almonds. The CRUD behaviour of each resource is dependent opon this key.
      • :almonds-tags: this key is an array, and is used to uniquely identify that resource. The above :almonds-type is also added to the tags array when the resource is added.
    • All other keys and values are those that correspond to the respective resource’s AWS Java API.
    • The two almonds properties are saved as AWS tags, when the resources are created remotely. These two properties are again added to the resources when they are retrieved from AWS, and are critical to state management.
    • References are also defined in terms of :almonds-tags (ex In the above example, the vpc-id contains the value of the :almonds-tags of the vpc). These references are resolved before performing any operations remotely.

    Below is an array of resources -

    (def my-resources [{:almonds-type :vpc
                        :almonds-tags [:sandbox :web-tier]
                        :cidr-block "10.2.0.0/16"
                        :instance-tenancy "default"}
    
                       {:almonds-type :vpc
                        :almonds-tags [:sandbox :app-tier]
                        :cidr-block "10.3.0.0/16"
                        :instance-tenancy "default"}
    
                       {:almonds-type :subnet
                        :almonds-tags [:sandbox :web-tier :web-server]
                        :cidr-block "10.2.11.0/25"
                        :availability-zone "us-east-1b"
                        :vpc-id [:sandbox :web-tier]}
    
                       {:almonds-type :subnet
                        :almonds-tags [:sandbox :app-tier :app-server]
                        :cidr-block "10.3.11.0/25"
                        :availability-zone "us-east-1b"
                        :vpc-id [:sandbox :app-tier]}])
    

    The above defines two vpcs and subnets, with each subnet having the vpc-id of the respective vpc.

    state management

    • almonds maintains three atoms in the memory for managing state -
      • local-state: this contains all the added resources. Whenever a resource is added it is added to the local-state.
      • remote-state: this contains only those resources that are avalaible remotely and have the almonds tags.
      • remote-state-all: this contains all the resources that are avalaible remotely - almonds resources or not (this state is helpful during library development for debugging issues)
    • the local-state and the remote-state are the source of truth. They both are used to determine the differential between the resources that are defined and the resources that exist remotely.

    staging resources

    • The almonds.api namespace contains the api.
    • When resources are added they are added to the local state.
    • Execution of the function on the REPL returns the :almonds-tags of all the resources that have been added.
    • The :almonds-type is added to the :almonds-tags vector of each resource.
    (require [almonds.api :all :refer])
    
    (add my-resources)
    
    
    ;; ==================>>>>>>>>>>>>>>>>>>>
    ;;
    ([:subnet :sandbox :app-tier :app-server]
     [:subnet :sandbox :web-tier :web-server]
     [:vpc :sandbox :app-tier]
     [:vpc :sandbox :web-tier])
    ;;
    ;; =====================================
    

    diff

    • When the diff is run, it returns a differential between the local-state and the remote-state.
    • It returns a hash-map with three keys -
      • :only-in-local: these are the resources which have only been added are not present remotely.
      • :only-on-remote: these are the resources which are not added but are present remotely (Remember the state is transient, and if you added the resourced from an REPL, and then created them, they will not be present in the staging state the next time you restart your REPL)
      • :inconsistent: these are resources which are present in both the staging state and synced state and also do not match.
    • If the remote-state is empty then the pull function is first called, which populates the remote-state by retrieving resources from the remote end.
    (diff-tags)
    
    ;; ====================>>>>>>>>>>>>>>>>
    ;;
    {:inconsistent (),
     :only-on-remote (),
     :only-in-local
     ([:sandbox :app-server :app-tier :subnet]
      [:sandbox :vpc :app-tier]
      [:web-tier :sandbox :web-server :subnet]
      [:web-tier :sandbox :vpc])}
    ;;
    ;; ====================================
    
    (diff-tags :sandbox :vpc)
    
    ;; ====================>>>>>>>>>>>>>>>>
    ;;
    {:inconsistent (),
     :only-on-remote (),
     :only-in-local ([:sandbox :vpc :app-tier]
                 [:web-tier :sandbox :vpc])}
    ;;
    ;; ====================================
    
    (diff :app-tier)
    
    ;; ====================>>>>>>>>>>>>>>>>
    ;;
    {:only-in-local
     ({:almonds-tags [:subnet :sandbox :app-tier :app-server],
       :almonds-type :subnet,
       :availability-zone "us-east-1b",
       :vpc-id [:sandbox :app-tier],
       :cidr-block "10.3.11.0/25"}
      {:almonds-tags [:vpc :sandbox :app-tier],
       :almonds-type :vpc,
       :cidr-block "10.3.0.0/16",
       :instance-tenancy "default"}),
     :inconsistent (),
     :only-on-remote ()}
    ;;
    ;; =====================================
    

    Convention: All results of evaluation are presented as - ;; ==>

    api functions

    • All api functions are varaidic and can take zero to n number of tags.
    • All api functions have two variations ex - diff and diff-tags
      • diff: displays the result in terms of the resource
      • diff-ids: displays the resource in terms of the resource-ids
    • The ids variations are a convenience, and can be utilized when its not necessary to view the full resources.

    sync-resources

    • The sync-resources function first performs a diff, and then calls the create and delete functions for the respective resources.
    • The sync-resources function like other in the api can also be invoked with specific
    • The resources under the :inconsistent key are not affected.
    • The pull function is called after the respective resources have been added/deleted.
    (sync-resources :app-tier)
    
    ;; ====================>>>>>>>>>>>>>>>>
    ;; the  below is printed on the console -
    ;;
    ;; Creating :vpc with :almonds-tags [:vpc :sandbox :app-tier]
    ;; Creating :subnet with :almonds-tags [:subnet :sandbox :app-tier :app-server]
    
    ;; ====================================
    

    modifying resources

    • When an existing resource is changed locally or remotely it will appear under the :inconsistent key.
    • In the example below the :cidr-block of both the vpc and subnet have been changed.
    • The diff shows both of these under the :inconsistent key.
    • Below they are recreated using the recreate function.
    (def app-tier [{:almonds-type :vpc
                    :almonds-tags [:sandbox :app-tier]
                    :cidr-block "10.4.0.0/16"
                    :instance-tenancy "default"}
    
                   {:almonds-type :subnet
                    :almonds-tags [:sandbox :app-tier :app-server]
                    :cidr-block "10.4.0.0/26"
                    :availability-zone "us-east-1b"
                    :vpc-id [:sandbox :app-tier]}])
    
    (add app-tier)
    
    (diff)
    
    ;; ====================>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
    ;;
    {:only-in-local (),
     :inconsistent
     ({:almonds-tags [:subnet :sandbox :app-tier :app-server],
       :almonds-type :subnet,
       :availability-zone "us-east-1b",
       :vpc-id [:vpc :sandbox :app-tier],
       :cidr-block "10.4.0.0/26"}
      {:almonds-tags [:vpc :sandbox :app-tier], :almonds-type :vpc, :cidr-block "10.4.0.0/16", :instance-tenancy "default"}),
     :only-on-remote ()}
    ;;
    ;; =================================================
    
    (recreate :app-tier)
    
    ;; ====================>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
    ;;
    ;; Deleting :subnet with :almonds-tags [:subnet :sandbox :app-tier :app-server]
    ;; Deleting :vpc with :almonds-tags [:vpc :sandbox :app-tier]
    ;; Creating :vpc with :almonds-tags [:vpc :sandbox :app-tier]
    ;; {:almonds-tags [:vpc :sandbox :app-tier], :almonds-type :vpc, :cidr-block 10.4.0.0/16, :instance-tenancy default}
    ;; Creating :subnet with :almonds-tags [:subnet :sandbox :app-tier :app-server]
    ;; {:almonds-tags [:subnet :sandbox :app-tier :app-server], :almonds-type :subnet, :availability-zone us-east-1b, :vpc-id [:vpc :sandbox :app-tier], :cidr-block
    ;;
    ;; =================================================
    

    recreating resources

    • There are four different ways in which the above could have been achieved
      • (recreate :app-tier): calling the function without the tag would have recreated all the resources.
      • (recreate-inconsistent): this will run the diff first and the recreate all resources that are inconsistent. If a tag is used then, then a diff will be run with the tag, thus limiting which inconsistent resources are recreated.
      • (delete-resources :app-tier) (add app-tier) (sync-resources): this will first delete the :app-tier resources, then add them and then create them.
      • (expel :app-tier) (sync-resources) (add app-tier) (sync-resources): this will remove the resources (remove them from local-state), then sync deletes them, then stage the resources, and then pull creates them.
    • The higher level functions are a combination of the more granular functions, however the granular ones can be used as needed.

    acknowledgements

    almonds uses the amazing amazonica library to interact with the AWS Java SDK. Its rapid development would not have been possible without it and also thanks to its maintainers for rapidly addressing issues raised during the dveloment of almonds.

    a big shout out to the whole clojure community, without which it would have been too cumbersome to write this tool.

    a big thanks to the emacs community which makes the process of development so productive and fun.