Architecture

Let’s start with the architecture of the solution.

NetFlow data from two firewalls is collected by the ElasticFlow collector. It’s a very convenient—and in many cases free—NetFlow/sFlow/IPFIX collector with built-in data export mechanisms, for example, to the ELK stack. ElasticSearch is used both to receive data from ElasticFlow and to make that data available to other applications—such as Kibana for visualization or my own script.

The LLM models run on a dedicated server with a suitable GPU. I run the models using Ollama software, which exposes an API for queries on the local network.

The script accepts a user query formulated in natural language. It then sends this query to the model to obtain a DSL filter structure in response. This filter is essential to extract only the records we’re interested in from Elasticsearch. The filter generated by the model is used in the Elasticsearch query. The response, which is in JSON format, is then parsed again by the LLM to translate it into natural language and display it to the user.

Elasticsearch query

Let’s start by looking at the Elasticsearch queries. The method query_es() defined in the ex_client.py module handles them. The communication is very straightforward because it uses the Elasticsearch API. We need to query the data available under a specified index or group of indices and narrow the query criteria using a DSL filter. A DSL (Domain Specific Language) filter in Elasticsearch is a specialized query language that allows you to define precise search and filtering operations on data within the indices. Its syntax is based on the JSON format, which makes the queries readable and easily integrated with other systems that use this popular data format. Here’s an example:

{
   "query":{
      "bool":{
         "must":[
            {
               "term":{
                  "destination.port":9200
               }
            },
            {
               "range":{
                  "@timestamp":{
                     "gte":"now-15m",
                     "lt":"now"
                  }
               }
            }
         ]
      }
   },
   "aggs":{
      "unique_hosts":{
         "terms":{
            "field":"source.ip",
            "size":1000
         }
      }
   },
   "size":0
}

Elasticsearch provides a well-documented API. Initially, I tried using the standard requests library to send and receive data via this API. Unfortunately, issues arose related to the formatting of keys in the DSL structure sent in the request’s body. The API required that key names be enclosed in double quotes, but the requests library converted them to single quotes. I quickly switched to the elasticsearch-py library, where API handling is implemented through dedicated methods. To establish a connection, I created the get_es_client() method. I pass only the information about the indices to be queried and the DSL filter to the search() method from this library. In response, I either receive the data or an error.

Requesting the LLM to generate a DSL filter

We retrieve the user input from the console. However, we need to provide the user’s input with some context in the query sent to the LLM model. Therefore, the user’s query is embedded in a larger prompt named ADAPT_ES_QUERY, defined in the prompt_templates.py file. This way, we give the query context and supplement it with additional data. This prompt also contains detailed information about the model’s expected result. In the program’s first iteration, I only passed the user’s query to the generated prompt, later expanding it with two additional pieces of information, which I’ll describe shortly.

The method generate_text() from the llm_client.py module is responsible for communication with the Ollama API. It uses the requests library to send a POST request, including in the payload information about which model to refer to and the query content. We also include additional flags there, such as disabling response streaming.

I use the same method and approach to analyze the response from Elasticsearch and translate it, in the context of the user’s query, into an answer that is understandable to them. For this, I only use a different prompt, the template for which is stored in GENERATE_FINAL_ANSWER.

First run

So, we now have what appears to be a ready-to-go product that works as follows:

• The user types their query into the console.
• The script asks the LLM model to translate the user’s query into a DSL filter for Elasticsearch.
• The script performs a query on the specified Elasticsearch indices using the obtained filter, defining the query parameters via the created DSL filter.
• The response from Elasticsearch, in JSON format, is passed to the LLM model so that it can translate the Elasticsearch answer into natural language in the context of the original query.

Easy, what can go wrong?

Problems with responses from the LLM

I tested generating responses on models such as Llama3, DeepSeek-R1 (with parameters 14B, 32B, and 70B), and qwen2.5-coder (with parameters 3B and 32B). I’ll discuss the significance of the parameters in a moment. The first problem was somewhat different. Some models wouldn’t return just the JSON structure of the DSL filter in their response. I’m referring primarily to DeepSeek-R1, which always ignored the request in the prompt to return only the JSON structure. Instead, its response always included the entire chain of the model’s “reasoning,” as if I had asked for an explanation.

The second problem was that some models sometimes did not return a clean JSON object as requested. For example, they would add additional Markup Language formatting. To address this, I added the method extract_json_from_llm_response() in the llm_client.py module, which is responsible for removing any extraneous information from the response. This method is called within generate_text().

A quick note here – remember, this is proof-of-concept code. The llm_client.py module should ideally include much more exception handling, data validation, etc.

Solving the first problem revealed the second, more serious issue. It turned out that the models were not generating correct DSL structures. This incorrectness manifested in two ways. First, the model would “invent” field names from the NetFlow data to use in the filters. For example, when querying for traffic on a destination port, it might sometimes use the field “destination.port,” other times “port,” or even “destination_port.” Other variations would often appear. The second problem was the incorrect structure of the DSL filter itself. Known keywords such as “aggs” or “size” were placed in the wrong parts of the structure, rendering the entire construction invalid.

How does the LLM work?

This brings us to the theoretical part, explaining why this happens. I should note right away that the information in this paragraph is somewhat simplified to be understandable even for those who don’t know how such models operate—without delving too deeply into the details.

Incorrect Generation of DSL Filters May Stem from Several Factors. Three Basic Ones Are:

Lack of Precise Training Data and Specific Patterns:
Language models are trained on massive datasets that might not contain sufficiently detailed and uniform examples of DSL queries for Elasticsearch—especially for specialized use cases like NetFlow traffic analysis. As a result, the model “hallucinates” field names and may arrange the query structure incorrectly, placing key elements in the wrong locations. This is due to the lack of strict supervision over the precise syntax and structure of the DSL, causing the model to improvise based on inconsistent or incomplete patterns found in the training data.

Model Limitations in Understanding Domain Structure:
LLM models generate text based on the probability of subsequent words. However, without an explicit mechanism to validate the structure of the resulting code or query, they can make both syntactic and semantic errors. The absence of a validation mechanism during generation means that even if the model “knows” which keywords should be used, it does not always correctly position them within the hierarchy required by the DSL.

A Model Chosen with Too Few Parameters:
In theory, increasing the number of parameters can enhance the model’s ability to capture complex patterns and relationships in the data, which would theoretically translate into more precise responses. However, a larger number of parameters also brings challenges, such as higher computational requirements, increased susceptibility to “hallucinations,” and difficulties in precisely controlling the generated text in very specific domains. Conversely, a model with fewer parameters may have a limited ability to model very complex dependencies, which can also affect the precision in generating specialized DSL structures. However, it might be more “conservative” in its improvisation. The numbers like “70B” or “32B” in a model’s name refer to the number of parameters in the model (70 billion and 32 billion, respectively).

Tuning the LLM

Issues with the LLM can be addressed in several ways. The first idea was to switch to using models from OpenAI or Azure—specifically those not hosted locally on my server. Such models returned much more correct data, particularly regarding the DSL structure’s correctness. They seem to be a bit better fine-tuned. However, the project’s idea was to use a local model.

The second option is to enrich the prompt sent to the LLM. That’s why, in the ADAPT_ES_QUERY template, I pass two additional pieces of information. As base_query_template, I provide an example DSL filter structure as a pattern. This is meant to help the model construct the response more accurately. The sample filter is stored in a file within the repository.

The second parameter, index_mapping, is a mapping of fields in the Elasticsearch index. This helps the model use the correct keywords. The index is several megabytes and is fetched from ElasticFlow via an API.

Introducing these changes made the query to the model significantly larger, which in turn requires much more resources to handle. This resulted in a slight improvement in the data returned by the model, but not enough to say that the problem has been fully solved.

The model performs much better when translating the Elasticsearch response into natural language—provided that the response contains the data the user asked for. Again, we face the issue that this is a proof-of-concept project, and proper mechanisms for verifying the Elasticsearch response should be implemented. Otherwise, the model’s interpretation of empty or incorrect data may generate an invalid answer for the user.

Further tuning ideas

This leads us to the section titled “What Else Can Be Done.” I haven’t tested these ideas yet:

• Using a Properly Trained Model:
  In my tests, I was limited by the performance of the server and GPU running the LM models and the availability of the models themselves. Models like OpenAI’s o1 seemed to generate better responses than Llama3. I also tested the recommended model qwen2.5-coder. This model has been fine-tuned on large datasets containing code and technical documentation, enabling it to generate coherent, correct, and contextually appropriate code in various programming languages such as Python, JavaScript, Java, C++, and others. As a result, the model can create new code snippets and assist in debugging, refactoring, or analyzing existing solutions. Unfortunately, I did not notice a significant difference between it and, for example, DeepSeek-R1.
• Searching for a Model Trained for Working with Elasticsearch:
  Unfortunately, I haven’t found a publicly available model already trained on data specific to communicating with Elasticsearch. Whether such a model exists in a free or commercial form, I do not know. Will one be developed? That’s not for me to say—but Elasticsearch producers are paying close attention to the use of AI, so something like that might emerge.
• Fine-Tuning the Chosen Model Yourself:
  Attempting your own model fine-tuning is also an option. However, this comes with the necessity of having training data and preparing it appropriately. Training a model does not simply involve dumping a text file into it and expecting it to magically learn how to work with the data. I also do not have the appropriate training data, as I don’t work with Elasticsearch daily.
• Modifying Query Parameters:
  When using the API interfaces of language models, you can pass some parameters that affect how the responses are generated. These parameters are not visible in web interfaces. For example, temperature determines the level of “randomness” in the generated text. A value close to 0 makes the model generate more predictable and deterministic responses, whereas higher values (e.g., 0.8 or 1.0) increase the variety of outcomes, which might lead to more creative but sometimes less consistent responses. top_p (nucleus sampling) sets the probability threshold within which the model selects the next tokens. For example, a value top_p=0.9 means that the model will choose from tokens whose total cumulative probability amounts to 90%. This parameter helps control the distribution of probabilities and can affect the creativity of the generated text. In this way, you might try to control the level of “creativity” in the generated responses.

In the project repository, I have included a correct DSL filter for querying which devices in the network have communicated on port 9200 during the last 10 minutes. By running the script with the --override-dsl parameter, you can check for yourself how much the LLM model’s generated response deviates from the expected outcome when asking an appropriate question. You can also see how the LLM model interprets the Elasticsearch response. Additionally, running with the --debug flag will display a lot more information on the screen—not just error messages.

Summary

This small project showed me that I can write a simple chatbot in one evening and the limitations of current LLM models. Remember that working with them through a chat interface on a website is somewhat different from using them programmatically. The project was successful because it demonstrated that such a chatbot could be built.

And what did I learn in the process? Well, that’s for me to know!

The post Ask about Netflow data using natural language appeared first on IT Playground Blog.

If you want to test the vWAN by yourself, or you are interested in how to automate the vWAN management using the Terraform (my favorite tool lately), check the Szkola DevNet’s (my DevNet education project for the polish speaking audience) GitHub, where I put complete Terraform script. Using it, you can create a simple vWAN topology with two hubs, a VPN connection to another VNet, some VNet Peering, and of course, routing.

https://github.com/SzkolaDevNet/Terraform-Azure-vWAN

The post Using Terraform to manage Azure vWAN appeared first on IT Playground Blog.

Why install own SSL certificate?

It is not essential to install own SSL certificate. Nothing prevents you from leaving the default certificate signed with an internal CA called VIRL Labs CA. The only thing we’re going to face is browser warnings that CA is not trusted. Also, running any scripts that use the API will require that you ignore invalid certificates. And what if such a server we would like to issue to the Internet to remotely manage simulations? Or write a phone application for such management? Then the built-in certificate will certainly not pass.

A few years ago, for the needs of my lab and home network, I launched the Certificate Authority service. I also created a set of scripts for generating keys and signing requests. I installed CA certificates as trusted on my devices. Thanks to this, on the one hand, I am sure that I connect to my device, on the other hand, I do not have to bypass security mechanisms built into the API.

If you want to build a similar infrastructure at your place, you can follow the description that I posted on my English-language blog a few years ago.

Let’s create the SSL certificate

In CML 2.0, certificates are used in two places. Those are the Lab Manager listening on port 443 and System Maintenance Controls on port 9090. I only exchanged the certificate in the first one because it is responsible for access to the GUI and API interface. It seemed to me that it would be a relatively simple operation and possible to be performed by the GUI or CMS administration panel. Nothing could be more wrong! Unfortunately, this is another thing that has not been thought of in CML 2.0 and requires a combination of things. Seriously, the form in the administrative panel would be enough. Even my almost 10-year-old NAS has it ….

We need to generate an encryption key, CSR (Certificate Signing Request), and sign it in our CA infrastructure. We can create the key and CSR on a machine with CML 2.0, but it will simplify your task. All we need is a computer with Linux and OpenSSL. We also need to know the IP address and FQDN (if we have given it) of our virtual machine with CML 2.0. For me, they are 172.16.1.5 and cml.virl.lan respectively. The key and CSR generating command looks like this:

SAN = DNS: cml.virl.lan, IP: 172.16.1.5 openssl req -new -nodes -out cml.virl.lan.csr -keyout cml.virl.lan.key -subj "/CN=cml.virl.lan "-config etc / server-SAN-env.conf

In my network, all certificates must have the Subject Alternative Name extension, hence the need to define the SAN variable and use the appropriate CA service configuration. I have not tested whether the certificate without SAN will work correctly in CML 2.0. We sign the CSR request generated this way using our CA.

openssl ca -config etc / signing-ca.conf -in cml.virl.lan.csr -out cml.virl.lan.crt -extensions server_ext

In this way, we generated the key and our CA certificate. We can proceed with the installation.

Installing new SSL certificate in CML 2.0

If there is no dedicated form used to change the certificate, we must manually replace it. Unfortunately, the SSH service in CML 2.0 will connect us with CLI to manage simulations, not the operating system. We have two alternatives. First, we can log in to the system administrator account using the virtual machine console. The second way is to call the command line from the System Maintenance Controls panel. Then we need to use for example SCP to copy the certificate and key files to the virtual machine operating system with CML 2.0.

The CML 2.0 interface is launched using the nginx. The configuration files are located in the /etc/nginx directory. Among them, the file with the key (privkey.pem) and certificate (fullchain.pem). The easiest way to back up these files and upload your own key and certificate that you created earlier. Let’s remember that to perform this operation, we need to increase our privileges using sudo.

Finally, we return to the System Maintenance Controls panel, go to the Services tab and find The nginx HTTP and reverse proxy server service. We enter its settings and restart. After restarting the service, we will see that it identifies itself with our certificate. If the restart fails, the reason information should be displayed on the console screen. For me, to my surprise, nginx did not require that the service certificate be accompanied by the certificates of CA itself and Signing CA.

The post How to change the SSL certificate in CML 2.0 appeared first on IT Playground Blog.

This article has been originally posted on my polish blog Szkoła DevNet (DevNet School)

CML 2.0 installation

The first thing that catches the eye is how you install CML 2.0. First of all, it is not possible to install the Personal version as a native operating system (bare metal). In the package, we only get a virtual machine image in OVA format and an ISO file containing images of virtual network devices that we need to connect to the created virtual machine. A bit unusual, but it doesn’t bother me, it only hinders the potential of moving a virtual machine between servers. Moving, because at the moment neither Personal nor Enterprise versions support clustering. VMware products are the only supported hypervisor. Home users don’t have to worry – Player, Workstation or Fusion versions are enough. Installation and commissioning take only a few minutes.

Of course, CML 2.0 does not work without a license – we need to download the token from our account on cisco.com, enter it in the appropriate field, and activate it.

The GUI

The entire CML 2.0 interface has been rewritten. To manage the system itself, we get a graphic System Maintenance Controls console that listens on port 9090. The simulation GUI is available on port 443. The VM Maestro application has been completely withdrawn, and we manage the simulation through a browser using a transparent HTML5 interface. We can see the level of resource consumption assigned to the virtual machine on an ongoing basis, managing simulations is straightforward, adding devices also. And unfortunately, the positive impressions end here.

In my opinion, the person responsible for interface design should be sent to the course about how to create graphical application interfaces. Here are the major mistakes made when creating the interface:

• The mouse on each device has at least two buttons. It is a mystery to me why the context menu is not implemented in the GUI. Clicking with the right mouse button on an object or workspace is a natural reflex of every person working at the computer. Many actions that now require a few clicks, for example, opening a connection to the console, could close in two. No context menu is hugely frustrating.
• Creating a connection between devices is extremely unintuitive. You need to select the device, click the small icon above it and hold to drag the connection, and finally choose the ports. It is so unintuitive and inconsistent with the entire interface that I had to look into the documentation on how to perform this essential task.
• All device parameters or connections are entered in a dedicated window at the bottom of the simulation. You could be a pop-up context window, instead yo you have to move your mouse a lot.
• Contextual hints are poorly designed, and it is often impossible to read their contents.

API in CML 2.0

Luckily, CML 2.0 also has an API. It allows you to do all the work bypassing the GUI. Well-structured API documentation built using Swagger is available. Thus, directly from the documentation, we can check the operation of specific methods. A Python library is also available. We can install it both from the .whl file available in the CML 2.0 installation and from the PyPi repository. Let’s always pay attention to the version – currently, in PyPi we will find an older edition of the library.

In Ansible Galaxy we will find modules for Ansible, which will allow us to create playbooks. I hope that now after separating the modules from the main part of the Ansible project, which I wrote about recently, Cisco will submit the modules to the official collection.

Where is the configuration generator?

He’s not here. And this is next to the little functional GUI my biggest surprise. At Cisco VIRL, one of the biggest advantages was that you could quickly generate a working topology. We set up devices, connected them, set parameters such as AS number, routing protocol, VRFs individually or for groups and clicked one button. Interface IP addresses and many parameters were generated automatically. All you had to do was run the simulation and you had a fully functioning test network.

In CML 2.0, the automatic building of the initial configuration is limited to giving descriptions to the interfaces, their activation, and cisco/cisco user settings. And that’s it.

I don’t know what the idea was for the creators. One of the best features of VIRL has been removed from the product. Of course, you can write the configuration, you can write scripts, you can use the provisioning tools thanks to the API, only this takes time, and not everyone has such knowledge.

Summary

Maybe I will change my mind, but after 4 hours of work with the CML 2.0 I am … maybe not totally disappointed, but very disappointed. I expected a step forward, and I did not know exactly what. A bit like a product that was developed without any customer satisfaction survey. I do not suspect that I would be any special and unusual customer.

There are still some functionalities that I haven’t thoroughly checked out and looked promising. Among other things, better integration of external solutions, or just the use of API. I will write about them soon. However, so far, my rating is 5/10.

CML 2.0 costs just like VIRL $ 199 for an annual subscription that can be purchased here.

The post Why CML 2.0 is quite a disappointment appeared first on IT Playground Blog.

Lately, I worked more and more with Docker Swarm clusters and wanted to automate some tasks. Unfortunately, with the Ansible 2.7 there were only three modules available – thedocker_swarm, docker_swarm_serviceanddocker_secret. The tasks I wanted to automate required features as reading the Swarm configuration (this was partially covered by the docker_swarm), node status and change the node configuration. I had three options – use the shell module and execute the CLI command locally on the remote host, wait for someone to provide the module or write the module by myself.

I may not be a professional developer, but I had a great teacher when I started coding in the last class of primary school and some experience in programming. You can’t really be an engineer if you cannot write scripts or simpler apps. I think it is not a surprise I decided to test myself in professional project

In the previous post, I presented the docker_swarm_facts module. I am a co-author there. Let me now show you the Swarm Node related modules fully developed and now maintained by me. All of them will be available in Ansible 2.8 release in next few weeks, but you can test it using the devel branch code on GitHub!

Tell me about my Docker Swarm node…

Before you start changing the configuration of any device, software etc. you really should get and store some information about its current state. To read the essentials about the Docker Swarm node you need to use the docker_node_facts module. The module inherits all default docker modules options like the TLS configuration. You can execute it locally using the Docker socket or remotely using the Docker API. Remember that Docker by default use an only local socket so you need to reconfigure your daemon to support remote management. And one of the most important requirements – you need to run the module on Docker Swarm Manager node.

    - name: Get the swarm node facts
      docker_node_facts:
        self: yes
        docker_host: "tcp://172.10.10.10:2376"
        tls: true

By default, Ansible will open an SSH connection to the remote host and execute the module there connecting to local Docker socket. The host is usually an inventory entry. Using the API requires providing the URL of the management interface as the docker_host parameter. You can execute the module still on the remote host (the same as in docker_host or different), but usually, in such case, the playbook is set with the parameter connection: local and run on the localhost. Understanding this difference is crucial for running any Swarm module and avoid problems.

You also need to tell the module information on which Swarm node you want the manager to return. The default behavior is returning facts about all Swarm nodes. If you know the names of the nodes you are interested in you can provide them as a list in the name parameter. Setting the self: yes option will tell the module you want facts only about the node module communicates with.

Module output matches the output of the docker node inspect CLI command. The nodes key in the returned structure contains an array of dictionaries where each element matches the CLI command. Its structure may vary depending on the version of Docker daemon and Docker API – you need to check the documentation for details.

Changing the node configuration

The docker_node module allows changing some parameters of the Swarm node configuration like the node role or the availability. The first parameter defines if the node is a manager or a worker. Initially, you must define the role when you add a node to the cluster. The availability defines how the swarm cluster will use the node for work distribution. Three allowed states are active (accepting new containers), pause (operating the existing containers but not accepting new ones) or drain (not accepting new containers and existing will restart on another node). The last state is useful during upgrades or when you want to remove a node from the cluster

The docker_node module also allows changing the labels assigned to the Swarm node. Labels are usefull for management and automated work distribution. Labels are the key-value pairs. The default module behavior is merging the labels provided in playbook with those already assigned to node and update the value of a label if it is already defined.

- name: Merge node labels and new labels
  docker_node:
    hostname: mynode
    labels:
      key: value

To override default behavior and replace the labels with new ones you must set labels_state: replace. To remove all assigned labels you also have to use this option, but without providing any new labels.

- name: Remove all labels assigned to node
  docker_node:
    hostname: mynode
    labels_state: replace

If you want to remove specified labels you need to provide the list of label keys in parameter labels_to_remove.

The post Manage the Docker Swarm nodes – new Ansible 2.8 modules developed by me! appeared first on IT Playground Blog.

Lately, I started missing some features in Ansible that will allow me to perform some operations on Docker Swarm clusters. And I try to avoid using the command or shell modules as much as possible – running CLI commands on a remote host is like asking for troubles. I decided to fill this gap by myself and as a result, I can say I am now the Ansible community developer, author, and maintainer of a few modules – docker_swarm_facts, docker_node_facts, docker_host_facts, docker_node; and co-author of docker_swarm and the ansible.docker.swarm library.

In next posts, I wanna show you how those modules work. However, if you are looking forward using them I strongly advise you to give them a try now and report any bugs you find so we can fix them before Ansible 2.8 release. You can find them in the devel branch of Ansible repository.

Docker Swarm management in Ansible 2.8

The first modules for Swarm features were introduced in Ansible 2.7. The docker_swarm is for Swarm cluster configuration and reconfiguration while docker_swarm_service is for services management. In Ansible 2.8 you will find lots of bug fixes and improvements for those modules. My addition to the swarm management modules family is docker_swarm_facts.

Reading the information about the Swarm is part of the docker_swarm already. However, this is against the Ansible design. According to the best practices separate modules should exists for reading the state and performing the changes. Of course in the second case module can return some information about the state after the change but should not have the inspect feature implemented. The dedicated modules for reading the state, or as it is called in Ansible the facts, should end with the _facts suffix.

My docker_swarm_facts module is designed for reading the swarm configuration form the Swarm manager node.

- name: Get info on Docker Swarm
  docker_swarm_facts

The output will contain the Swarm cluster information including the worker and manager Join Tokens. It also gathers Swarm parameters available in output of other CLI commands, not only Swarm related ones. Using the module you can easily get the Raft parameters, Cluster ID or Lock status. It returns also Swarm configuration version numer which is not accessible via CLI.

List Swarm objects

There is one additional feature allowing you to get the lists of the nodes, services, and tasks. The result includes names as well as necessary information similar to the output of docker node ls, docker service ls or docker service ps commands. You can easily use those lists in other tasks in loops.

- name: Get info on Docker Swarm and extended list of registered nodes
  docker_swarm_facts:
    nodes: yes
    tasks: yes
    services: yes

To request list you need to set dedicated parameter to yes. Default output contains almost the same information as the respective CLI command output. To get more details you can add verbose_output: yes parameter which extends the range of attributes for each object on the list. However it is still not as complete as dedicated _facts module output for each object type.

- name: Get info on Docker Swarm and extended list of registered nodes
  docker_swarm_facts:
    nodes: yes
    verbose_output: yes

For each type of listed objects you can also use filters.

Happy testing of Ansible 2.8!

The post Using Docker Swarm? You gonna love Ansible 2.8! appeared first on IT Playground Blog.

The central point of my home and lab networks is the Juniper SRX220H – it is quite old but the services and its performance fulfills my requirements so far. One of the services it provides is DHCP Server for some of the VLANs. I was not really missing the domain-search feature so far but decided to add it to make my work more convenient. This feature configuration is not straightforward, and there are many misleading guides about setting up this feature on different platforms or operating systems. Let me show you how it works on Juniper.

What is the domain-search?

The domain-search is a DHCP protocol feature to provide the client list of the domain suffixes to search if only the hostname is known. It is different than domain-name which points just to the primary domain. The domain-name is the first position in the domain-search list. Both of them are the implementation of the DHCP Options (check the link here).

The domain-search is an RFC-standardized DHCP Option 119. If you ever worked with DHCP Options you know it can be tricky. My router runs on 12.1X46 firmware (the latest for this hardware) which supports the JDHCP CLI – this is the standard one in newer JunOS lines. However, the JDHCP CLI does not support domain-search directly via the dedicated configuration option. It was quite simple with the older JunOS CLI

[edit system services dhcp]
set domain-search [ lan virl.lan ];

Using the modern approach the DHCP server configuration is part of the access address-assignment section. But there is no command to set the domain-search parameter, so we have to configure it straightforward as Option 119

The misleading docs

The RFC3397 states that Option 119 is “a string specifying the search list”. What it does not say is the value is a hex-string, but there is a little hint in the example in this RFC document. If you google a little you will find lots of examples dedicated to Cisco or Windows. You can see examples with string encoded into hex-string. You will also find there useful link to on-line converter (I put it at the end of this article). Use it to convert the list of domains into hex-string. For me, the “lan virl.lan” after conversion is “6c616e207669726c2e6c616e0d0a”. Lets put it into the configuration.

set access address-assignment pool LAN family inet dhcp-attributes option 119 hex-string 6c616e207669726c2e6c616e

If you now capture packets on the client device you can see the response is malformed

We can see the text containing list of provided domains in raw packet output but it is not recognized by the client.

Special characters

Lets get back to the example in RFC3397 describing how the DHCP Option 119 works.

   Below is an example encoding of a search list consisting of
   "eng.apple.com." and "marketing.apple.com.":

   +---+---+---+---+---+---+---+---+---+---+---+
   |119| 9 | 3 |'e'|'n'|'g'| 5 |'a'|'p'|'p'|'l'|
   +---+---+---+---+---+---+---+---+---+---+---+

   +---+---+---+---+---+---+---+---+---+---+---+
   |119| 9 |'e'| 3 |'c'|'o'|'m'| 0 | 9 |'m'|'a'|
   +---+---+---+---+---+---+---+---+---+---+---+

   +---+---+---+---+---+---+---+---+---+---+---+
   |119| 9 |'r'|'k'|'e'|'t'|'i'|'n'|'g'|xC0|x04|
   +---+---+---+---+---+---+---+---+---+---+---+

You will find the full explaination of this example in RFC. Lets focus on one aspect interesting us the most. Every Option 119 entry (first cell represents the option not the value) starts with number ‘9’. This numer must equal to a number of characters provided in the single Option 119 record. The limit of the single record is 255 characters, so we will not split in our configuration.

After the ‘9’ in the first record follows number ‘3’. This value must equal the length of the following part of the domain name. So the “eng.apple.com” must be encoded as “3eng5apple3com”. The checksum numbers replace the dot. Another special case is a ‘0’, which replaces the space (0x20) separating the domains in text configuration. And finally, there must be something closing the whole hex-string. In RFC it is ‘C004’ which is a pointer to use the ‘apple.com’ defined previously again, but for us, it will be a ‘0’.

We used the characters here to make the example more clear, but remember we have to provide the hexadecimal string as our record. Because each sign is encoded by hex duplet it is quite easy to split converted text and add the required special markers. There is one more thing – we don’t provide the first number, which tells about the lengths of each part of our configuration (the ‘9’ in the example). Therefor our command will be:

set access address-assignment pool LAN family inet dhcp-attributes option 119 hex-string 036c616e00047669726c036c616e00

Lets test it

Receipe and trick

As you can see it is working now. If you need a summary what to do here is a step-by-step recipe:

• Convert into hex string the list of domains as you put it into domain-search parameter usually. You can use this website. Remember not to start a new line in this form
• Split the output into individual hexadecimal numbers – every two characters in the output is one hexadecimal number.
• Replace the spaces (0x20) with 0x00, put the 0x00 at the end
• Replace the dots (0x2e) with 0x03
• Prepend each word with value equals to a number of characters in this word. Should be easy to identify them because now every word ends with either 0x00 or 0x03, but don’t include them in calculated number.
• Concatenate everything so you get a hex-string

However, if you want to provide more than 2 domains it may not be easy to calculate the proper value or update it. To simplify the work and make configuration more flexible instead of using the hex-string type we can use array hex-string instead. We list there each domain separately as next array elements.

set access address-assignment pool LAN family inet dhcp-attributes option 119 array hex-string [ 036c616e00 047669726c036c616e00 ]

The post DHCP domain-search on Juniper SRX appeared first on IT Playground Blog.

In some cases you may need to access the host resource from the container like the Docker own socket. It is required if you run Docker inside of the Docker container or you deploy a tool that will manage your Docker hosts or Docker Swarm cluster. The problem is the socket on the hosts is owned by the root, while the root PID from inside of your container is remapped to non-root PID on the host. It make the host socket inaccessible for the processes inside the container. But there is an easy workaround for this problem

Namespaces and containers

Unix socket is a special handler that allows applications to communicate. To make it simpler, in this case, think of it as of TCP connections, but instead of connecting to IP and port you connect to a socket on the host. To make it, even more, simpler think of the socket as of the file on the filesystem which you read from and write to sequences of bytes. The owner of the socket is the process creating it.

Using the user namespaces we face the situation where the dockerd process (the Docker Engine) is run as root while the containers itself uses non-root PID. The Docker socket permissions in such case should look like that

srw-rw----  1 root   docker     0 Jan 16 22:52 docker.sock

The process running in the container from the host perspective is a non-root process if we use the user namespaces

231072     512  0.0  1.4  20888 13856 ?        Ss   Jan16   0:16 /usr/bin/python2 /usr/bin/supervisord

The first column is the user ID – on my system it is 231072. If we bind the socket to the container it won’t be accessible because of the ownersip and permission mismatch

The socat

The solution to our problem is a small application – socat. It is available in most Linux distributions by default. It created two bidirectional pipes (byte streams) and transfer data between them. The endpoints can be by example the socket type of SOCK_STREAM. The other end might be…. another socket! And another process may be the owner of the socket.

srw-rw----  1 root   docker     0 Jan 16 22:52 docker.sock
srw-rw----  1 231072  231072    0 Jan 16 22:52 docker-userns.sock

The second socket can easily be assigned and accessible for processes inside the container.

Starting the socat we need to specify the both endpoints – first the new one following by existing root-owned socket.

# /usr/bin/socat UNIX-LISTEN:/var/run/docker-userns.sock,user=231072,group=231072,mode=0660,fork UNIX-CLIENT:/var/run/docker.sock

And thats it!

The post Accessing host socket when using namespaces for Docker isolation appeared first on IT Playground Blog.

The vertical scaling means adding more workers doing the same process in parallel, but tasks assigned to each process should not overlap. In my case, I wanted to spread the polling across my RaspberryPi cluster. Polling process can consume the CPU and make RaspberriPi unresponsible. In worst scenario, if you have many devices or you are poling lots of data from your devices the polling processes may not finish their work within 5 minutes (this is how often devices have to be queried) so you will miss some data. You may of course tune number of threads the polling process start but any platform has it resources limitations you cannot bypass.  

In this article I will present my solution based on Docker Swarm cluster. It has some limitations and downsides but may work for many people, not only on RaspberyPi.

Observium poller vertical scaling idea

The idea of poller vertical scaling has its own section in Observium Community Edition documentation. Unfortunately, one crucial paragraph is… empty! But I will cover it here. Make sure you get familiar with the official documentation before the further reading of my article because I will only slightly focus on some configuration aspects described in the documentation.

You will need the vertical scaling in two cases:

• When host running Observium is running out of resources
• When network segmentation requires multiple discovery and polling agents because you cannot access all devices from single place in your network.

In both cases, you can separate the polling and discovery from the web front-end. However, all of them will still need access to the database and RRD repository. 

In my lab, the problem was that polling took over 4 minutes at some point (old devices are slow to respond) and consumes a significant amount of CPU making some of the RaspberryPi 3 B processes unresponsible. Therefore I wanted to distribute polling across whole Swarm stack leaving front-end and discovery in a single independent container. 

There are two things you need to remember when you try to build scalable Obervium infrastructure and distribute the services across multiple hosts:

• The recurring tasks are the cron jobs. You need to run them exactly as defined in configration templates. Running them more often or not often enough will corrupt gathered data. 
• The recurring tasks run on separate hosts cannot overlap each other. Running discovery or housekeeping on each poller node means wasting the resources. If pollers on different hosts query same device you create situation similar to running poller to often on single hosts which may corrupt the data

Front-end web server, database and RRD’s

The database is hosted on my NAS and it has to be accessible by all Observium instances no matter of its role. Observium configuration file contains all required details as hostname, port, credentials, and database name.

The rpi-1 host runs single container with Observium CE and RRDCached service. The RRDCached is a daemon that receives updates to RRD files and writes them into the files. It allows multiple hosts send updates at the same time taking care of the writing to disk process. You can find the rrdcached package in most Linux distributions. If you want to use it in Observium you need to update the configuration file and uncomment NETWORK_OPTIONS=”-L” line. This will tell the daemon to listen on TCP port 42217 for incoming updates. You cannot use the RRDCached and update the RRD files locally at the same time, so once you configure the daemon you must update the Observium configuration file. Inside the container you can’t use the socket to write to rrdcached daemon, you must use the network connection.

In my lab, the RRDCached and web front-end (including the all recurring tasks except the poller) share the same container. In large installations, it will be good to separate them – that will let you scale the web front-end service as well using the docker swarm replicas. However, the container with web front-end cannot perform housekeeping and discovery anymore.

Poller scaling (static)

This paragraph covers the missing section in Observium documentation.

As I described before the idea of vertical scaling is to distribute the polling process across multiple devices. Each poller instance should query predefined group of devices in such way, that each device recorded in Observium database is queried by exactly one poller every 5 minutes. To achieve this you need to provide two additional parameters to poller-wrapper.py script – the total number of pollers running in your system (-i) and the index number unique for each poller (-n). Using those two values poller script will determinate which devices from the database it should query.

As much as this approach is easy it creates the serious problems in scaling. The poller-wrapper.py is the script defined in the cron configuration and each poller container must have the cron configuration with correct parameters. Every time you add or remove the poller container, you must update the cron configuration of all other poller containers. It can cause problems no matter if you use stand alone containers or Swarm.

I decided to go with static configuration files. I use Docker Swarm stack to achieve the load distribution and redundancy, so I always have 4 replicas of poller containers. Technically it is exactly one replica of four different containers, because each must have dedicated cron configuration provided.

ID                  NAME                             MODE                REPLICAS            IMAGE                                                                        PORTS
smh1yzx3nwc3        observium-poller_poller-node-0   replicated          1/1                 observium-ce-poller:ubuntu19.04   
dvzli7cr5765        observium-poller_poller-node-1   replicated          1/1                 observium-ce-poller:ubuntu19.04   
4cplgqyrqxrz        observium-poller_poller-node-2   replicated          1/1                 observium-ce-poller:ubuntu19.04   
59bytcl7pb0o        observium-poller_poller-node-3   replicated          1/1                 observium-ce-poller:ubuntu19.04   

There are several way you can provide the configuration file for each container. I am using the configs section introduced in the docker-compose configuration file version 3.3. Because each Swarm worker node must have access to the configuration file I store them on remote NFS share mounted to each node. If I change number of workers I need to update all configuration files, but at least I have them in one place.

configs:
  observium-poller-node-0.cron:
    file: /home/docker-nfs/observium-poller-swarm/observium-poller-node-0.cron

services:
  poller-node-0:
       configs:
         - source: observium-poller-node-0.cron
           target: /etc/cron.d/observium-poller

The same way I also provide the Observium configuration file (config.php) because it is common for all containers. The downside is that every time you edit configuration files you must update the Swarm stack which means rebuilding all containers. If this is an issue in your network then attach the configuration files to containers in volumes sections. Any change will be visible in the container right away. Cron checks the configuration files every minute and you don’t need to restart its process or the container to apply changes.

Dynamic poller scaling (ideas)

The presented approach is not what you may expect thinking about the vertical scaling using containers and Docker Swarm stack. It is more load distribution and redundancy than scaling, but it fits my requirements at the moment. The main problem in adapting it to dynamic scaling is the cron configuration that must contain a correct unique pair of parameters for each replica. As far as I looked in the Docker Swarm documentation there is no API method, command or environment variable you can use to read the number of configured and active replicas. Parsing the console output should always be the last resort solution. And it may be even better if you can read them from the inside of the container. But even if you get those two values you still need to dynamically update the cron configuration in each replica.

At this moment I don’t know any flexible and reasonable solution using tools available in docker or external free software that would let me solve those problems.

• Script reading number of replicas from CLI output on the manager, calculating the instance number for each running replica and those values as environment variables to each container. This can be tricky as well. You can provide the total number of replicas to all containers using the docker service update command, but to provide unique instance number for each instance you may need to use docker exec to set it directly. Then you need another script running inside the container that will periodically recreate the corn configuration based on actual environment variables values.
• You may use the database to maintain required information. The minimum information you need to store is container name and assigned instance number you will later use to generate the cron configuration. You need to write the script that will register the container, read the instance number value and regenerate the cron configuration. This script will run inside each container. Other script running on stack manager must still update all of the service containers on number of total running replicas.

If you know how to solve this issue easier let me know, I will test it and describe it.

The post Observium pollers vertical scaling appeared first on IT Playground Blog.

VMware Harbor runs in containers which simplifies managing the software, but the upgrade is not straightforward as you may think. Most significant change is database consolidation. Instead of separate databases for Harbor, Clair and Notary version 1.6.0 introduces a single database engine for all components – the PostgreSQL. The 1.5.0 Harbor uses the MySQL, while Clair already uses the PostgreSQL. The developers prepared a dedicated container with migration engine that performs all the work. However, I found upgrade documentation missing the crucial explanation of steps and commands which may lead to loss of your data. I will try to cover my findings in this post.

Make your own backups

The first problem I found is that documentation does not clearly explain if you should refer to configuration file from the old or new version. It does not even explain that in some aspects upgrade scripts update the configuration file. So before you start doing step described in the documentation, do the following:

You are now safe having own manual backup in case you run migration tool with wrong parameters.

Database backup

The migration process should be simple because developers created a container with dedicated software and scripts performing all operations for us. It is available on Docker Hub at goharbor/harbor–migrator. You need to download containers using the proper tag, but the documentation is not clear which one. You can choose between the v1.5.0 and v1.6.0 tags. Use the tag matching the final version of your upgrade! If you select the container with the tag matching your current VMware Harbor version the scripts will not work and error messages will not be clear on that.

Then the documentation asks you to backup the database/harbor.cfg file to some backup location. If you have no such file just ignore this step. I could not find the configuration file in other locations than root project folder.

Now you need to run the harbor-migrator container to make a database backup. It mounts three volumes defined in docker run command but again the documentation is not clear on exact directories you should refer to. Instead, you find some system variables you probably never defined on your host. Replace the variables or define them using following values:

1. ${harbor_db_path}:/var/lib/mysql – directory of current version of database, by defailt it is /data/database
2. ${harbor_cfg}:/harbor-migration/harbor-cfg/harbor.cfg – location of the configuration file from the old Harbor version
3. ${backup_path}:/harbor-migration/backup – anywhere on host disk where you want to save the backup files generated by the script in container. It will make copy of the harbor.cfg files and dump the MySQL database into registry.sql file

Database migration

In the next step, you will perform the actual migration of the database using the same harbor-migrator software in a container. The command requires the volumes defined in points 1 and 2 in the previous paragraph as its parameters. It will overwrite the content of the /data/database folder on the host with a new database. If you by mistake run the harbor-migrator container using the tag that matches your current VMWare Harbor version the conversion will not work and you may lose all files in /data/database directory. Usually, the script will end with the message that no migration was performed for Harbor version defined in the harbor.cfg file.

If the database migration completes successfully your configuration file will be updated with a new _version variable definition. Remember, it changed the file in the folder of your old Harbor version, not the one you unpacked few minutes ago.

If you use the Clair or Notary modules you must migrate their databases separately using again the harbor-migrator software. You will find the commands to execute in the documentation. Again you need to mount the volumes with an old and new database but surprisingly this time documentation refers to explicit paths instead of some variables.

The last task you will use the harbor-migrator is moving the updated configuration file to the directory with new Harbor version files. However, instead of running the script in the container again you can just copy the harbor.cfg file manually. If you followed steps I described earlier copy harbor.cfg from folder harbor-1.5.1 to folder harbor.  

Recreate the configuration files

In the last step described in the upgrade manual, you use the install.sh script to generate required configuration files. If you want to use Clair, Notary or Chartmuseum you need to run the script with additional parameters as described here. It will create the new configuration files in common/config subfolder. 

If you use a modified version of the docker-compose configuration files you must update the default ones found in the harbor archive you downloaded previously. I have changed mine by upgrading them to version 3. Also to connect Harbor with Jenkins containers I run on the same host I defined a dedicated subnet which I had to include in my configuration.

If installation script runs with no problems it will pull containers of all VMware Harbor components from the Docker Hub repository (or local archive if you decided to perform the offline installation) and run containers. 

Summary

Upgrading the VMware Harbor is not a hard task. Everything, even the migration scripts, runs in containers – you just download new images and recreate the containers after all. Database migration might be tricky because the documentation lacks some details in my opinion. In most production or test containers setups you do not refer to system variables in docker-compose configuration files. For security reasons, everything except the credentials should be hardcoded. 

The post Upgrading the VMware Harbor appeared first on IT Playground Blog.