Archive of articles classified as' "Programming"

Back home

How to monitor cronjobs in kubernetes with sk8l

8/12/2023

Installation

To achieve our goal, we need to install 3 tools: sk8l, prometheus and grafana. We will continue in that order.

sk8l installation

You can install sk8l via helm.

helm repo add sk8l https://sk8l.io/charts
helm repo update

helm search repo sk8l

helm show values sk8l/sk8l

helm upgrade --install [RELEASE_NAME] sk8l/sk8l \
--set namespace.name=[NAMESPACE] \
--set serviceAccount.metadata.namespace=[NAMESPACE]

sk8l configures the pod and containers as secure as possible so there are some settings that you might want to take a look at and configure if neeeded via the cli or by creating your own values.yaml.

  • namespace:
    You can create a namespace where you want to run sk8l in via the helm chart with namespace.createNamespace. If you chose to, you can pass namespace.labels in case you want to enforce security policies at a cluster-level to ensure all pods in the namespace meet the security standards via the the PSS and PSA controllers.
  • networkPolicy.uiPolicy:
    sk8l-api only allows traffic from the containers running the user interface.
  • networkPolicy.apiserverPolicy: The sk8l-api pod needs to be able to reach the api-server and therefore you need to restrict the values for ingress.from.ipBlock.cidr and egress.to.ipBlock.cidr appropriately.
  • configMaps.ui.vue_app_sk8l_api_url: The frontend needs to know how to reach envoy. The default value is localhost, so you only need to change this if you want to reach sk8l on another address.
  • secrets:
kubectl create secret tls -n NAMESPACE tls-ca-cert --cert=ca-cert.pem --key=ca-key.pem  
kubectl create secret tls -n NAMESPACE tls-server-cert --cert=server-cert.pem --key=server-key.pem

After configuring and installing the chart, sk8l should be seeing an screen similar to the following running at https://localhost:8001.

If there are no cronjobs currently configured to run on the same namespace as sk8l, thet that page will look very empty with only a hint message.

You probably want to see sk8l in action, so add an example cronjob with the following code:

Now that you have a cronjob configured and while you wait for some jobs to be done, you can click around to see all the information sk8l displays about a cronjob and job executions.

This is pretty cool(if i say so myself) but cooler is to see all the data available on some charts. For that we need to configure prometheus and grafana and that’s what we’re going to do next.

Prometheus

There are different ways of how to install prometheus and how to do it depends on how you want to run it. In this section I’m going to give you some pointers on what needs to be configured and how to do it in case you chose to run it inside kubernetes.

TLS certificates

sk8l exposes a /metrics endpoint on an HTTPS port, therefore you need to make the server and ca certificates available to prometheus too in order for it to secure the communications between both applications.

Secret management is a big topic and there are multiple ways to handle them in your applications. Since we created the tls certificates above as secrets directly via kubectl, we are going to keep doing the same here. We create a deployment and mount a volume with both files to the pod running prometheus.

NetworkPolicy

As mentioned above, sk8l tries to be as secure as possible. Every connection is rejected by default, this is why we need to allow explicitly every connection that needs to take place to and out of its pods. In other words, we have to add a policy to whitelist connections coming from the prometheus pod on their corresponding namespace to sk8l-api on the scrape port(8590).

Scrape jobs

You need to tell prometheus what applications to scrape metrics from. This is done via a prometheus.yaml file on the machine prometheus runs on. We are going to add a ConfigMap to k8s to create a file with the scrape jobs we need to get cronjob metrics out of sk8l.

Once prometheus is running with this configuration, it should be reachable at http://localhost:9090 and you could query and even graph data as completed, failing, registered cronjobs coming from sk8l on http://localhost:9090/graph, just type sk8l or the namespace you’re using on the search field.

Grafana

We continue now with grafana, which will help us see all the data in better looking dashboards.

No Comments

Run a remote ruby script locally.

25/11/2012

If you ever want to try a ruby script without downloading it you can do it like this:

ruby -e "$(curl -fsSL https://raw.github.com/gist/3862719/bb2a7a18f3dc4d95cfa77d637317e296a114eak1/some_script.rb)"

I know this is not rocket science but i want to have this line handy.

No Comments

rake routes filtered by controller

23/09/2011

rake routes CONTROLLER=controller_name

No Comments

Paperclip content type validation always fails on Internet Explorer

23/09/2011

I just ran into a strange problem. It turns out that when you try to upload a file using paperclip on IE, the content-type is different from other browsers and you have to take of them if you want to use paperclip’s validates_attachment_content_type method.

I needed to add to my model validation the following content-types: “image/pjpeg” and “image/x-png”. I did not have a problem with gifs.

validates_attachment_content_type :logo, :content_type => ['image/jpeg', 'image/jpg', 'image/gif', 'image/png', 'image/pjpeg', 'image/x-png']

1 Comment

Rails 3.1: CoffeScript & Asset Pipiline screencasts

7/09/2011

Rails 3.1 – CoffeScript from BostonRB on Vimeo.

Rails 3.1 – The Asset Pipeline from BostonRB on Vimeo.

No Comments

rbx-require-relative requires Ruby version ~> 1.8.7

17/08/2011

If you ever run into this problem when trying to run bundle install using ruby 1.9.2 all you need to do is change in your Gemfile ruby-debug to ruby-debug19.

Answer @ Stackoverflow

2 Comments

Disruptive Innovation

5/07/2010

Because companies tend to innovate faster than their customers’ lives change, most organizations eventually end up producing products or services that are too good, too expensive, and too inconvenient for many customers.

By only pursuing “sustaining innovations” that perpetuate what has historically helped them succeed, companies unwittingly open the door to “disruptive innovations”.

Disruptive Innovation

No Comments

Rails: perdiendo la cabeza con before_save

9/12/2009

Recientemente encontré un ‘bug’ en una aplicación que estoy desarrollando, tiene que ver con los callbacks de Rails, en específico before_save.

Por fín lo he resuelto, no de la manera que esperaba, tal vez por berrinchudo ya que aún me queda una duda, pero al fin puedo continuar desarrollando y un peso se ha ido de mí.

La parte afectada de la aplicación involucra 2 modelos, digamos:

class Category
  has_many :products
end
 
class Products
  belongs_to :category
end
 
app/models/category.rb
before_save
  return unless self.published
  self.products.first.amount = (algun calculo dinámico)
end

Justo en este callback es donde está lo misterioso, la cantidad(amount) del primer producto nunca cambiaba amenos que actualizara la categoría, la publicara y la grabara 2 veces!! a la 2a vez el cálculo se realizaba, esto me quitó el sueño y me hizo perder gran cantidad de tiempo analizando el problema.

por qué a la 2a vez si pasaba?

Corrí la aplicación con el debugger y pues Rails sí pasaba por esa parte del código pero la cantidad del producto quedaba intacta, incluso intente cosas como estas:

app/models/category.rb
before_save
  return unless self.published
  self.products.first.amount = (algun calculo dinámico)
  self.name = 'Algún nombre'
end

Y pues la primera vez que grababa, ya con el atributo de published en true, name sí se actualizaba pero amount no! sólo hasta la segunda… hasta la fecha no entiendo por que sucede esto, no he logrado llegar a una conclusión, lo que sí es que por lo pronto ya encontré una solución:

app/models/category.rb
before_save
  return unless self.published
  self.products.first.update_attribute(:amount, (algun calculo dinámico))
end

De esta manera el cálculo ocurre todas y cada una de las veces que la categoría es actualizada.

No Comments