Panther: Name service cache

This article is written in English and Portuguese
Este artigo está escrito em Inglês e Português

English version:

A recent thread in the IIUG mailing list (relative to a reverse DNS issue) reminded me of a new Panther (version 11.7) functionality that was on my list for articles. I've been avoiding many of the bigger and more important features because they will take a lot of time to write about... I hope this one will be shorter.

Informix needs to read several files or interact with DNS servers each time you try to open a connection. Considering Unix and Linux (Windows is a bit different technically, but not that much conceptually), these are some of the actions the engine must do:

1. Depending on your host resolution criteria it will probably open the /etc/hosts file to search for your client's IP address. If it's not there it will contact your DNS server in order to request the name associated with the IP address.
   Note that all this is done by a system call.
2. It will access /etc/passwd (or equivalent) to get your user details (HOME dir, password - this is probably stored in another file like /etc/shadow - , user id, group id etc.)

The engine must also access /etc/services and /etc/group in other situations.
Depending on your environment these activities can take a bit of time, and require some significant CPU usage. There are systems with high number of connections per second which can naturally transform this into a significant issue.
To give you an example I do regular work on a system that used to receive a very large number of requests from CGI programs. So, each request received by HTTP required a new process on the application server, and a new connection on the database server. They had peaks of tens of requests per second. Currently they're using Fast CGI with noticeable improvements.
Anyway, IBM decided to give us the chance to optimize this by caching previous results (file searches and DNS requests). This is done with the new parameter called NS_CACHE (from Name Service Cache). The format of this $ONCONFIG parameter is:

host=num_secs,services=num_secs,user=num_secs,group=num_secs

Each comma separated pair (functionality=num_secs) configures the number of seconds that the engine will cache a query result for that functionality. I'm calling it functionality, because it can be implemented through files or system APIs. The documentation could be clearer, but let's check each one:

• host
  This is the host and IP address resolution service. Depending on your system configuration (on most Unixes and Linux this is specified in /etc/nsswitch.conf) it can be resolved by reading the /etc/hosts file and/or making a request to your DNS servers
• service
  This should be the map between service names and system ports, usually done by reading /etc/services. The only situation that comes to my mind where this is used is when you're trying to start a listener (either during engine startup or after that with onmode -P) or when you're trying to make a distributed query to another engine, and you use names in your INFORMIXSQLHOSTS instead of port numbers. In any case, I may be missing something...
• user
  This is very important. It refers to all the user related info that Informix gathers from the OS and that is relevant to Informix. The information can be stored in /etc/passwd, /etc/shadow, or indirectly be managed by external services like LDAP. It can include:
  - Home dir
  - User ID
  - Group ID
  - Password
  - User status (enable or disable
  
• group
  This relates to the OS group information. Usually done by reading /etc/group

If the specified number of seconds to cache the information is zero, it means that we don't want to cache it. So the behavior will be the old engine behavior (for each relevant request, the information must be obtained).
The parameter can be changed online with onmode -wm

It's important that you fully understand the implications of caching these kind of information. By asking Informix to cache this info, we're also assuming the risk of working with stale information. Let's imagine a simple scenario. Assume this sequence of events:

1. At time T0 you connect using user and password to the engine which is setup to cache user information for 600s (10 minutes).
2. At time T1 you change that user password
3. At time T2, the same user tries to connect to the Informix database with the new password. It will fail!
4. At time T3 (T0 + the number of seconds to cache user information) the user repeats the connection attempt with the new password. It will succeed!

How can you avoid situation 3? If you change the cache timeout to 0, it will work as a flush.
If for example you do some changes to your user's information you can run:

onmode -wm NS_CACHE="host=900,service=900,user=0,group=900"
onmode -wm NS_CACHE="host=900,service=900,user=900,group=900"

These commands will flush the user information cache, and then reactivate it.

So, the point I'd like to make is that this feature can help you improve performance (specially for systems with an high connection rate), but it can have some side effects. You can workaround these ones, but for that you must know they exist.


Versão Portuguesa:

Uma discussão recente na lista de correio do IIUG (relativa a um problema com reverse DNS) lembrou-me de uma funcionalidade nova do Panther (versão 11.7) que estava na minha lista de temas a abordar. Tenho andando a evitar muitas das maiores e mais importantes novidades porque vou demorar bastante tempo a escrever sobre elas.... Espero que esta seja mais reduzida.

O Informix tem de ler diversos ficheiros ou interagir com servidores de nomes (DNS) cada vez que abre uma conexão. Considerando o Unix e Linux (em Windows será um pouco diferente tecnicamente, mas não muito conceptualmente), estas são as acções que o motor tem de fazer durante o estabelecimento de uma conexão:

1. Dependendo do critério usado para resolver endereços e nomes, provavelmente irá abrir o ficheiro /etc/hosts para procurar o IP da conexão. Se não o encontrar irá provavelmente contactar o servidor de nomes (DNS) e pedir o nome associado ao IP de onde chega a conexão.
   Note-se que isto é feito com uma chamada de sistema e não cabe ao Informix definir os critérios.
2. Irá aceder ao /etc/passwd (ou equivalente) para obter os dados do utilizador (HOME dir, password - isto deve estar guardado noutro ficheiros como o /etc/shadow- , id de utilizador, id de grupo etc.)

O motor também tem de aceder ao /etc/services e /etc/group noutras situações.
Dependendo do seu ambiente estas operações podem demorar um pouco e requerer um consumo de CPU relevante. Existem sistemas com muitas conexões novas por segundo o que naturalmente pode transformar isto num problema sério.
Para dar um exemplo, trabalho regularmente com um sistema que em dada altura recebia um enorme número de pedidos por CGI. Sendo CGI, cada pedido recebido via HTTP requeria um novo processo na máquina do servidor aplicacional, e uma nova conexão na base de dados. Tinham picos de dezenas de ligações por segundo. Actualmente estão a usar Fast CGIs com benefício notórios.
De qualquer forma a IBM decidiu dar aos utilizadores a oportunidade de optimizarem estes aspectos, através de uma cache que guarda respostas anteriores (pesquisas em ficheiros e resultados de DNS). Isto é feito com um novo parâmetro designado NS_CACHE (de Name Service Cache). O formato do parâmtro do $ONCONFIG é:

host=num_segs,services=num_segs,user=num_segs,group=num_segs

Cada par (funcionalidade=num_segs) separado por vírgula, configura o número de segundos durante os quais o motor irá manter em cache o resultado de uma pesquisa para essa funcionalidade. Estou a chamar-lhe "funcionalidade", porque pode ser implementada usando ficheiros ou APIs de sistema. A documentação deveria ser mais clara, mas vamos ver cada uma:

• host
  O serviço de resolução de nomes e endereços IP. Conforme a configuração do seu sistema (na maioria dos Unixes e Linux isto é definido em /etc/nsswitch.conf) pode ser resolvido pelo ficheiro /etc/hosts ou fazendo um pedido aos servidores de DNS
• service
  Este é o mapeamento entre o nome de serviços e as portas de sistema, habitualmente feito através da leitura do ficheiro /etc/services. As únicas situações que me ocorrem em que isto é usado é quando arrancamos com um listener (seja no arranque do motor ou depois quando se usa o onmode -P), ou quando tentamos executar uma query distríbuida a outro motor, e usamos nomes no nosso INFORMIXSQLHOSTS em vez de números de portos. Mas pode estar a escapar-me alguma coisa, e haver outras...
  
• user
  Este é muito importante. Refere-se a toda a informação relativa aos utilizadores que o Informix obtém do sistema operativo e que é relevante para o Informix. A informação é guardada no /etc/passwd e /etc/shadow, ou gerida indirectamente em serviços externos como LDAP. Pode incluir:
  - Home dir
  - ID de utilizador
  - ID de grupo
  - Palavra passe
  - Estado do utilizador (activo, inactivo)
  
• group
  Isto diz respeito à informação de grupos do sistema operativo. Normalmente feito por consulta ao ficheiro /etc/group
  
  

Se o número de segundos especificado para a cache for zero, significa que não queremos fazer caching. Portanto o comportamento será o antigo do motor (para cada pedido a informação tem de ser obtida).

O parâmetro pode ser modificado online com o comando onmode -wm

É importante que entenda completamente todas as implicações de fazer caching deste tipo de informação. Ao pedir ao Informix que guarde e reutilize a informação já obtida, estamos também a assumir o risco de trabalhar com informação entretanto desactualizada. Vamos imaginar um cenário simples. Consideremos a seguinte sequência de eventos:

1. No momento T0 conectamo-nos usando um utilizador e palavra chave a um motor configurado para efectuar caching por 600 segundos (10 minutos).
2. No momento T1 mudamos a palavra chave desse mesmo utilizador.
3. No momento T2 o mesmo utilizador tenta conectar-se ao Informix usando a nova palavra chave. Vai falhar!
4. No momento T3 (T0+ o número de segundos configurado para a cache de utilizador) o utilizador repete a tentativa de acesso com a nova palavra chave. Vai ter sucesso!

Como pode evitar a situação do ponto 3? Se mudar o tempo de cache para 0, funciona como uma limpeza da cache.
Se por exemplo efectuar mudanças na informação dos utilizadores, pode executar:

onmode -wm NS_CACHE="host=900,service=900,user=0,group=900"
onmode -wm NS_CACHE="host=900,service=900,user=900,group=900"

Estes comandos fazem a limpeza da informação e depois re-activam a cache.

Portanto, o ponto que gostaria de frisar é que esta funcionalidade pode melhorar o desempenho (especialmente em sistemas com elevada frequência de novas conexões), mas também pode ter efeitos secundários. Estes podem ser contornados, mas para isso temos de saber que existem.

Read more »

Posted in: cache,conection,conexão,dns,informix,performance,reverse dns,service

OAT: One Admirable Tool

One first warning: I'm still lacking time to write the kind of articles I like, which are focused, technical and relatively deep.
Having said that I feel better proceeding with yet another "chat like" article. And in this I would like to write about Open Admin Tool, or simply OAT.

I had two great sessions in IIUG 2010 conference, which although not specifically about OAT, were somehow related to it. I'm talking about John Miller session about Auto Update Stats, and Hyun-Ju Vega session covering SQLTRACE. Unfortunately I had to miss the session about OAT itself from Erika Von Bargen (i was presenting at the same time). Now, why did I like this sessions so much? The speakers were great but they were not the reason. The reason is fairly simple: They changed the way I thought about some things and they made me look into OAT again. I don't recall having written many things about OAT, but to be honest I though it was nice, but maybe more directed to the new people starting with Informix. Long time Informix DBAs obviously have a pack of scripts for the trivial stuff: running update statistics (I wrote my own scripts based on previous work from Informix staff and obviously we all know dostats from Art Kagel), check some performance issues (sessions with more CPU usage, tables with full scans, tables with more read/writes etc.), historical data from the instance (profile counters, table access stats etc.).

So, a nice tool, much better than previous attempts like Informix Server Administrator (ISA), nice looking but maybe for "newbies". Honestly this was more or less in my mind, even though I did not have full conscience about it.

Then I assisted this two sessions. And during the sessions we were guided through lots of "accessories". John showed several tasks, the task scheduler, the intelligence behind the auto update stats, and eventually even without noticing it, he effectively showed the audience how easy it is to work with OAT. Then Hyun-Ju, guided us through a real case scenario where SQLTRACE was key to solve a customer issue. Again, OAT was not the main focus, but the ease of usage and how we can take advantage of SQLTRACE in OAT SQL Explorer again surprised me.

Then a real life scenario came... Today, while working on a customer, I was asked to help the DBA team to understand a big memory consumption from some sessions in a development instance. The developer team really didn't noticed anything strange, since the engine was doing what was expected. But the DBAs noticed some sessions occasionally grabbed around 8-10 times the maximum memory usually consumed by most of their sessions. The more or less strange thing was that the session allocated the memory but when we got to see that, the memory was mostly free (although still allocated to the session). So, no strange pool usage was evident. Since my job their is also to try to pass some knowledge, I showed them how to setup SQLTRACE for one specific user and we asked that user to run the job(s). After lunch we had everything explained in the SQLTRACE buffers. The onstat -g his could be used to get evidences, but I decided to give OAT a try. And there it was, in a graphical manner all the session steps. We noticed that they had a terrible amount of recursive procedures (actually proc_a calls proc_b and proc_b calls proc_a) inside a foreach loop. So a lot of stack was necessary to hold the context during execution and so the memory was allocated and then became free.

Could I have made this with other tools? Probably. Lots of onstats, scripts etc. Would it be so easy? Not even in my best dreams!

The true magic about OAT is not OAT itself. It's how OAT glues several new features and makes them usable. Think about some things we can do in OAT:

1. Manage and use SQL history
   
2. Manage all the engine programmable tasks
3. Generate performance and usage reports
4. Change the engine configuration
5. Optimize storage
6. Manage your cluster and replication

All these have something behind that you frequently see in the "new features" section of the release notes (using the same order as the previous list):

1. SQLTRACE
2. Database scheduler
3. Automatic tasks and internal engine profiling
4. SQL API
5. Compression
6. MACH 11

I could have made the list bigger. The point is that probably for the first time in Informix history we have a tool that matches the "features". This has always been one of the most criticized (what was probably fair) aspects of Informix. But since v11.10 and the beginning of OAT we see that OAT has been upgraded fixpack after fixpack to match the engine new functionalities. This shows it's not another tool that tries to keep critics away, but is in fact a fundamental piece of the engine. And it follows the engine "DNA": Simple to install, simple to use, reliable and light.

Is it perfect? Probably nothing is... I think one area or aspect could be improved. And here I'm just echoing a customer thoughts. OAT should be able to provide a view and working mode for enterprise wide environments. Meaning some of it's features should be aware of all the instances configured in a customer environment. Some simple examples:

• A single alarm viewer page for all the instances (maybe "all belonging to one group")
• The monitors could collect/centralize into a common repository
• The monitors should have self conscience of the instances in a MACH-11 cluster

Some of these points and others I don't mention here could be provided by:

• Replicate each instance captured data to a common repository (either by ER or simple tasks)
• Create a plugin for showing data in that centralized repository
• Change some of the monitors to INSERT also the server name and direct the data to the primary server (for read only secondaries) using for example the connection manager

So, this limitations can be solved by customers. But it would be nice to see some improvements in the future.
I'd like to end the article sending my compliments and appreciation to the OAT development team and leave here a few links to help you understand what's behind OAT:

• Open Admin Tool home
  http://www.openadmintool.com/
  
• Manage your Informix database with the IDS OpenAdmin Tool
  http://www.ibm.com/developerworks/data/library/techarticle/dm-0807kudgavkar/
• Build plug-ins for IBM's OpenAdmin Tool for Informix Dynamic Server
  http://www.ibm.com/developerworks/data/library/techarticle/dm-0808vonbargen/
  

Read more »

Posted in: informix,monitoring,OAT,Open Admin Tool,performance,SQLTRACE