k8s-deployer now can be configured through env variables, added suppo…

…rt for multiple services in the spec file, fixed several minor bugs

Dockerfile

@@ -1,7 +1,7 @@
-# docker build --no-cache -t k8s-deployer:0.1 .
-# docker run -it -d --name k8s-deployer -P k8s-deployer:0.1
+# docker build --no-cache -t k8s-deployer .
+# docker run -it -d --name k8s-deployer -P k8s-deployer
 
-FROM alpine:3.5
+FROM alpine:3.6
 
 ENV SRV_PORT=8089 \
     PROJECT_DIR=/data/k8s-deployer

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2017 Milos Buncic
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -57,16 +57,32 @@ supervisorctl add k8s-deployer
 ```
 
 #### Docker
+Supported environment variables
+
+**Note:** Environment variables have precedence over configuration file
+
+| Env Keys                         | Env (default) Values | Value Examples                               | Description                                              |
+|:---------------------------------|:---------------------|:---------------------------------------------|:---------------------------------------------------------|
+| K8S_DEPLOYER_KUBE_SCHEME         | http                 |                                              | Scheme http or https                                     |
+| K8S_DEPLOYER_KUBE_HOST           | localhost            |                                              | Kubernetes API hostname or IP address                    |
+| K8S_DEPLOYER_KUBE_PORT           | 8080                 |                                              | Kubernetes API port                                      |
+| K8S_DEPLOYER_KUBE_API_HEADERS    | none                 | key1\_\_value1,key2\_\_value2,keyN\_\_valueN | HTTP request headers                                     |
+| K8S_DEPLOYER_CONSUL_SCHEME       | http                 |                                              | Scheme http or https                                     |
+| K8S_DEPLOYER_CONSUL_HOST         | localhost            |                                              | Consul API hostname or IP address                        |
+| K8S_DEPLOYER_CONSUL_PORT         | 8500                 |                                              | Consul API port                                          |
+| K8S_DEPLOYER_CONSUL_KEY_PATH     | kubernetes           | kubernetes/prod                              | Consul K/V store path where all the data will be stored  |
+| K8S_DEPLOYER_CONSUL_SPECS_RETENT | 5                    |                                              | How many specifications have to be preserved at any time |
+
 Build and run
 ```
-docker build --no-cache -t k8s-deployer:0.1 .
-docker run -it -d --name k8s-deployer -p 8089:8089 k8s-deployer:0.1
+docker build --no-cache -t k8s-deployer .
+docker run -it -d --name k8s-deployer -p 8089:8089 k8s-deployer
 ```
 
 
 Usage
 ---
-For quick test deploy you can use `deploy.sh` script located in examples dir, in this dir you will also find `echoserver.json` specification descriptor that will be used for deploying the `echoserver` service.
+For quick test deploy you can use `deploy.sh` script located in examples dir, in this dir you will also find `echoserver.json` specification descriptor that will be used for `echoserver` service deployment
 
 ```bash
 cd examples
@@ -93,10 +109,17 @@ Specification descriptor:
 }
 ```
 
-If you already have `yaml` spec files you can use `kubectl` to convert these locally to `json` format.
+If you already have `yaml` spec files you can use `kubectl` to convert these locally to `json` format
+
+**Note:** Only services of [type NodePort](https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport) will be registered in the Consul service catalog
+```bash
+kubectl convert -f echoserver-deployment.yaml --local -o json > echoserver-deployment.json
+kubectl convert -f echoserver-service.yaml --local -o json > echoserver-service.json
+```
+
+After successful `yaml => json` conversion you can use `k8s-specgen.py` script to easily generate `k8s-deployer` specification file
 ```bash
-kubectl convert -f echoserver-deployment.yaml --local -o json
-kubectl convert -f echoserver-service.yaml --local -o json
+k8s-specgen.py -d echoserver-deployment.json -s echoserver-service.json -o echoserver.json
 ```
 
 Kubernetes documentation regarding deployment and service objects
@@ -111,53 +134,56 @@ https://kubernetes.io/docs/reference/
 
 ---
 
-Insert specification for a new service into Consul K/V store
+Examples
+---
 
-**Note:** by default max 5 specifications for the same service will be held in the Consul K/V store (you can modify this value in the configuration file)
+In the following example we're going to explain how we can deploy `echoserver` service in `default` namespace
 
-**Note:** after successful transaction specification id will be returned as `Location` header value
-```bash
-curl -X POST -isSL -H 'Content-Type: application/json' --data '@echoserver.json' http://localhost:8089/specifications/default
-```
-or (default namespace value is `default` that's why we can omit namespace in URL path, unless we don't want to deploy a service in some other namespace)
+#### Insert specification for new service into the Consul K/V store
+
+**Note:** by default max 5 specifications for the same service will be preserved on the Consul K/V store at any time, you can modify this value in the configuration file or through environment variable `$K8S_DEPLOYER_CONSUL_SPECS_RETENT`
+
+**Note:** after successful transaction, specification ID will be returned as value of `Location` header
 ```bash
-curl -X POST -isSL -H 'Content-Type: application/json' --data '@echoserver.json' http://localhost:8089/specifications
+curl -X POST -isSL -H 'Content-Type: application/json' --data '@echoserver.json' http://localhost:8089/specifications/default/echoserver
 ```
 
-List all available specification ids
+#### List all available specification IDs
 ```bash
 curl -isSL http://localhost:8089/specifications/default/echoserver
 ```
 
-Show service specification
+#### Show service specification
 ```bash
 curl -isSL http://localhost:8089/specifications/default/echoserver/latest
 ```
 
-Deploy a new service using specification previously inserted into Consul K/V store
+#### Deploy a new service using specification previously inserted into the Consul K/V store
 
-**Note:** if we don't specify an id, `latest` specification will be used
+**Note:** if we omit specification ID, `latest` specification will be used
 
-**Note:** after every successful build `deployed` spec will be created in the `kubernetes/specifications/<namespace>/<servicename>` tree
+**Note:** after every successful build `deployed` spec will be created on this Consul K/V path `$K8S_DEPLOYER_CONSUL_KEY_PATH/specifications/<namespace>/<service_name>`
 ```bash
 curl -X PUT -isSL http://localhost:8089/deployments/default/echoserver
 ```
-or you can explicitly set specification id
+or you can explicitly set specification ID
 ```bash 
 curl -X PUT -isSL http://localhost:8089/deployments/default/echoserver/1490691025506482_1650b288-e79c-4247-9b3b-95f1051302c4
 ```
 
-Undeploy service
+#### Undeploy existing service
 
-**Note:** it's going to delete service from Kubernetes and service definition from Consul K/V store
-```
+**Note:** it's going to delete all the service related objects from Kubernetes and service definition from the Consul K/V store
+```bash
 curl -X DELETE -isSL http://localhost:8089/deployments/default/echoserver
 ```
 
 Update existing service definitions that have been manually modified on the Kubernetes side or
 populate Consul K/V store with new service definitions for services that are not deployed through `k8s-deployer` (register service on Consul)
 
-```
+**Note:** Deletion of services that are not fully deployed through `k8s-deployer` API will not be possible via API itself
+
+```bash
 curl -X PUT -isSL http://localhost:8089/registration/default/echoserver
 ```
 

consul-template/k8s-svcgen.py

@@ -16,16 +16,21 @@
 #
 # Now, restart the agent, providing the configuration directory:
 #
-#     $ consul agent -dev -config-dir=/etc/consul.d
+#     # consul agent -dev -config-dir=/etc/consul.d
 #     ==> Starting Consul agent...
 #     ...
 #     [INFO] agent: Synced service 'svc1'
 #     ...
+#
+# or
+#
+#     # consul reload
+#     Configuration reload triggered
+#
 
 import sys, json
 
-args = sys.argv
-data = args[1]
+data = sys.argv[1]
 
 svcs = []
 
@@ -36,6 +41,9 @@
         for s in ns.values():
             svc = json.loads(s)
 
+            if svc['spec']['type'] != 'NodePort':
+                continue
+
             name = svc['metadata']['name']
             annotations = svc['metadata'].get('annotations')
             node_port = svc['spec']['ports'][0]['nodePort']
@@ -58,4 +66,4 @@
             })
 
     services = {'services': svcs}
-    print(json.dumps(services, indent=2))
+    print(json.dumps(services, indent=4, separators=(',', ': ')))

examples/deploy.sh

@@ -2,5 +2,5 @@
 
 K8S_DEPLOYER_HOST="localhost:8089"
 
-SPEC_ID=$(curl -X POST -isSL -H 'Content-Type: application/json' --data '@echoserver.json' http://${K8S_DEPLOYER_HOST}/specifications/default | awk -F'/' '/^Location:/ {print $(NF-1)"/"$NF}' | tr -d '\r')
+SPEC_ID=$(curl -X POST -isSL -H 'Content-Type: application/json' --data '@echoserver.json' http://${K8S_DEPLOYER_HOST}/specifications/default/echoserver | awk -F'/' '/^Location:/ {print $(NF-1)"/"$NF}' | tr -d '\r')
 curl -X PUT -isSL http://${K8S_DEPLOYER_HOST}/deployments/default/${SPEC_ID}