Debugging Customizable Kickstart File

Kickstart file automates the installation Linux system. It does not provide a convenient way to pass parameters. Luckily,  passing kernel parameters to %pre installation section works. This method requires you to write your kickstart generation script in %pre section, then in the command section, you call the generated kickstart file.

Unfortunately, this mean it is more likely to fail if the parameters are not correct. How to debug? If you can at lease reach the installer, this document may point you direction.

1. Get the shell access

After anaconda the installer starts,  there are several console tabs, the ones that are useful are:

• main: the main information screen (Alt-F1 or Ctrl-Alt-F1)
• Shell: the shell access (Alt-F2 or Ctrl-Alt-F2)
• GUI: show the install progress if graphical mode is not disabled (Alt-F6 or Ctrl-Alt-F6)

For now, shell access is the focus. Switch to Shell tab by pressing Alt-F2 or Crt-Alt-F2.

2. Inspect parameters by looking at /proc/cmdfile

Just cat /proc/cmdfile to inspect whether the parameters looks right. Personally, I do not want to write complex parser, thus I use quote_plus() from python to encode tricky characters like space, line feed, $ and so on.

3. Inspect the generated scripts and log files

The first thing to check is generated kickstart file. Then check following interesting files in /tmp

• ks-script-XXX: The script in  section %pre  will be here. Inspect it, and even run it to see how it works or not
• *.log: various of log files

Enjoy the debugging. 🙂

Solved: nginx module incompatible

I encountered when I run nginx -t:

nginx: [emerg] module
"/usr/lib64/nginx/modules/ngx_http_auth_spnego_module.so"
version x instead of y in 
/usr/share/nginx/modules/mod-spnego-http-auth-nginx-module.conf:1

It indicated that the module version is incompatible.
If nginx was installed using yum or dnf, use following command to set the version:

yum downgrade nginx{,-all-modules,-filesystem,-mod-*}-[version]-[release]

Do not do the math with expr

expr EXPERSSION returns 1 when the EXPERSSION is empty or 0.  Thus, you cannot really use it to do arithmetic.

For example, you may find your script doing  expr 1 - 1  returns as error.

To be fair, man page expr actually mentions this. However, instead of a dedicate section EXIT STATUSES, it is hidden in the second last paragraph of DESCRIPTION.

So do yourself a favour, use BASH arithmetic like $((var+1)).

Chewable Fedora Atomic

Fedora Atomic is an operating system targets to containers. However, the documentation, for me, is very hard to read. In this document, I will share my adventure with Fedora Atomic, starting with post installation.

I have installed Fedora Atomic, now what?

Let’s say you want to install the tree and docker-compose for better understanding the directory structure of your new host; you also need to ping a host every hour; and of course, you want your shiny new docker application to be deploy to this machine.

Package Install with rpm-ostree

Install Google-Chrome in OpenStack RHEL 7 instances

We have automated tests that require runnable Google Chrome. Yet the Google Chrome kept crashing.

The first encountered is:

libGL error: failed to load driver: swrast
libGL error: Try again with LIBGL_DEBUG=verbose for more details.

This one is easy, install mesa-dri-drivers solved this.

Then cames:

ERROR:sandbox_linux.cc(338)] : InitializeSandbox() called with multiple threads in process gpu-process

My initial guess was SELinux, but journalctl returns nothing about it. After a few hours, I thought, how about firefox? Maybe it helps to set the SELinux and install the missing dependencies? And… Tada, both Firefox and Google Chrome worked. Eventually, I dug out that Google Chrome requires fonts to works. Specifically, liberation-fonts-common and liberation-sans-fonts.

To sum up, following command worked for me:

sudo yum -y install mesa-dri-drivers liberation-fonts-common liberation-sans-fonts

Update for ChromeDriver user

If you are also use ChromeDriver. Be aware that ChromeDriver 2.31 and up requires libc.so.6(GLIBC_2.18), yet RHEL 7 only provide libc.so.6(GLIBC_2.17)

Clamav: troubleshooting of clamdscan

clamdscan is much faster to run than clamscan, however, it requires clamd which is a bit harder to setup, so I have some tips for troubleshooting:

ERROR: Could not lookup : Servname not supported for ai_socktype

Usually you should check the permission, especially whether the current user is in group clamscan (the primary group of the clamd running user).

lstat() failed: Permission denied. ERROR

This is usually because clamd running does not have the permission to run the is run as non-root user.

So you will need to enlist User clamscan (the user that runs clamd). You need to logout and login to make that change effective.

If it is still failed with the same error messsage, it is still possible that you are fooled by ACL permission. Use getfact to check it. The reason? When you ls, you get:

drwxr-xr-x+ 2 testuser testuser 40 Jul 17 15:19 /tmp/test

But your actual ACL (getfacl /tmp/test)might look like:

getfacl: Removing leading '/' from absolute path names
# file: tmp/test
# owner: testuser
# group: testuer
user::rwx
group::---
mask::r-x
other::r-x

Acknowledge:

The Clamav image is from http://www.stepbystep.com/how-to-integrate-clamav-into-pureftpd-for-virus-scanning-on-debian-squeeze-45061/

Jenkins SSH: Invalid PEM structure, ‘—–BEGIN…’ missing

Quick Solution

1. Paste the RSA private key (e,g. content of ~/.ssh/id_rsa) to Jenkins, like this:
2. Ensure the ~/.ssh/known-hosts in Jenkins master has agent/slave host key. Like:

   stdbuf -o0 -e0 ssh-keyscan -H  &>> ~/.ssh/known_hosts

3. Ensure the ~/.ssh/authorized_keys of Jenkins agent/slave has Jenkins master’s RSA key.
4. (Optional) if you can access the terminal of Jenkins master, test the connection by using:

   ssh 

Why?

Because the issue Presence of ECDSA SSH keys breaks SSH credentials plugin seems to affect the Jenkins masters that have both ECDSA and RSA.

Jenkins: No entry currently exists in the Known Hosts file for this host

I have encountered the following error message when I was trying to connect Jenkins slave after plugin update:

[SSH] WARNING: No entry currently exists in the Known Hosts file for this host. Connections will be denied until this new host and its associated key is added to the Known Hosts file.
 Key exchange was not finished, connection is closed.
 java.io.IOException: There was a problem while connecting to xxx.xxx.xxx.xxx:22

I have tried to connect using ssh in console, it connected successfully, but Jenkins still refuse to connect.

Then I discovered that, if I provided the RSA Host key, Jenkins can now connected to slaves.

I guess the reason is ssh just use the known host key to determine whether it is known, and be able to fallback to RSA for actual authentication. On the other hand, Jenkins does not seem to have the fallback. You given RSA identity, then Jenkins expect RSA in known_hosts.

The issue is already filed as JENKINS-42959 Failed known_hosts verification for SSH agent. In the meantime you can use following workaround:

stdbuf -o0 -e0 ssh-keyscan -H <host> &>> ~/.ssh/known_hosts

The stdbuf here is for printing the stdout and stderr as the order of time they appear, just like what you would see in console. Otherwise the stderr will go first and then stdout.

Docker: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?

When I run

docker images

on my newly setup docker in Fedora 21, it shows:

FATA[0000] Get http:///var/run/docker.sock/v1.16/version: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?

The problem resolve after I re-login. So the error seems like that the permission is not yet applied, as I just add myself to docker group without logout.

So, if you cannot or don’t want logout just yet, use

sg docker "docker <subCommand> ..."

As long as you are in docker group in /etc/group, sg won’t ask you password.