Linux Troubleshootin Steps..!!

Linux login issue - Can not login: Resource temporarily not available

Sometime we get following error while user login:

$su - username
Coud not login: Resource temporarily not available

Most of the times this is because of processes or number of files available to user is already used. So, could not arrange more files/processes to login, in this case we required to increase the limits for user in /etc/security/limits.conf (PAM module)

username        soft    nproc           3000
username        hard    nproc          4096
username        soft     nofile           6000
username        hard    nofile          8192


Set these values as per requirement else setting up these values higher may result in server inaccessible in case that particular uses all file descriptors, process and higher memory of server.

Mount: /dev/sdb1 already mounted or /mnt busy

mount: /dev/sdb1 already mounted or /mnt busy

[root@]# mount -t ext3  /dev/sdb1 /mnt
mount: /dev/sdb1 already mounted or /mnt busy

lsof didn’t provide any open files that might be linked to this problem or there was any “famd” running. Finally doing the following steps to remove the logical devices from the device-mapper driver helped us fix the problem.
[root@]# dmsetup ls
ddf1_44656c6c202020201028001510281f033832b7a2f6678dab   (253, 0)
ddf1_44656c6c202020201028001510281f033832b7a2f6678dab1  (253, 1)

[root@]# dmsetup remove ddf1_44656c6c202020201028001510281f033832b7a2f6678dab1

[root@]# dmsetup ls
ddf1_44656c6c202020201028001510281f033832b7a2f6678dab   (253, 0)

[root@]# dmsetup remove ddf1_44656c6c202020201028001510281f033832b7a2f6678dab

[root@]# dmsetup ls
No devices found

 

How to Protect GRUB with Password in RHEL / CentOS / Fedora Linux

GRand Unified Bootloader (GRUB) is a default bootloader in all Unix-like operating system. As promised in our earlier article “How to reset a forgotten root password“, here we are going to review how to protect GRUB with password. As mentioned earlier post, anyone can login into single user mode and may change system setting as needed. This is the big security flow. So, to prevent such unauthorized person to access system we may required to have grub with password protected.

Here, we’ll see how to prevent user from entering into single user mode and changing the settings of system who may have direct or physical access of system.

Cautious: We urge to take backup of your data and try it out at your own risk.

How to set GRUB password after Linux installed

Login to the your machine as root from there type ‘grub’
#grub

As the user root, proceed as follows to set a boot password:

At the root prompt, encrypt the password using grub-md5-crypt:
# grub-md5-crypt
Password: ****
Retype password: ****
Encrypted: $1$lS2dv/$JOcHkLosIn7CJk9xShzzJVw/

[root@tecmint ~]# vi /boot/grub/menu.lst

OR

[root@tecmint ~]# vi /boot/grub/grub.conf

Note : I advise you to take backup of the files before making any changes to it, if in case something goes wrong you can revert it.

STEP 3: Add the newly created MD5 password in GRUB configuration file. Please paste copied password below timeout line and save it and exit. For example, Enter the line password –md5 <add the copied md5 string

[root@tecmint ~]# vi /boot/grub/grub.conf or [root@tecmint ~]# vi /boot/grub/menu.lst

# grub.conf generated by anaconda

#

# Note that you do not have to rerun grub after making changes to this file

# NOTICE:  You have a /boot partition.  This means that

#          all kernel and initrd paths are relative to /boot/, eg.

#          root (hd0,0)

#          kernel /vmlinuz-version ro root=/dev/sda3

#          initrd /initrd-[generic-]version.img

#boot=/dev/sda

default=0

timeout=5

password --md5 $1$TNUb/1$TwroGJn4eCd4xsYeGiBYq.

splashimage=(hd0,0)/grub/splash.xpm.gz

hiddenmenu

title CentOS (2.6.32-279.5.2.el6.i686)

        root (hd0,0)

        kernel /vmlinuz-2.6.32-279.5.2.el6.i686 ro root=UUID=d06b9517-8bb3-44db-b8c5-7710e183edb7 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=auto rhgb quiet

        initrd /initramfs-2.6.32-279.5.2.el6.i686.img

title centos (2.6.32-71.el6.i686)

        root (hd0,0)

        kernel /vmlinuz-2.6.32-71.el6.i686 ro root=UUID=d06b9517-8bb3-44db-b8c5-7710e183edb7 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=auto rhgb quiet

        initrd /initramfs-2.6.32-71.el6.i686.img

Reboot system and try it pressing ‘p‘ to enter password to unlock and enable next features

After rebooting the system and selecting the Linux entry from the boot menu, the following error message is displayed:
Error 32: Must be authenticated
Press Enter to enter the menu. Then press P to get a password prompt. After entering the password and pressing Enter, the selected operating system (Linux in this case) should boot.

 

 

 

HowTo: Recovering Linux Grub Boot Loader Password

If you have, a password protected grub boot loader and you forgot both root and grub password, then you can recover grub-boot loader password using the following method/procedure:

* Use Knoppix cd
* Remove the password from Grub configuration file
* Reboot the system
* Change the root password
* Setup new Grub password if required (optional)

Knoppix Live cd

The best way to recovering (or reset) a password (and to solve many other problem) is useKnoppix Live CD(http://knoppix.net/) . First downloaded knoppix and burn it on CD. Put CD into CD/DVD rom and boot the system. You need to boot system into a single user mode, at boot: prompt type linux single:
boot: linux single

Fig.01 knoppix in action

Mount Partition

You need to mount a partition where you grub configuration file is stored. Depend upon your Linux distribution the location of Grub configuration file changes. Following list summaries Grub configuration file location for pouplar distros:

§  Debian Linux Grub configuration file location: /boot/grub/menu.lst

§  Suse Linux Grub configuration file location: /etc/grub.conf

§  Red Hat/Fedora Linux Grub configuration file location: /boot/grub/grub.conf

If you forgot partition name use the fdisk -l command to obtained partition list / names:
# fdisk -l

Fig.02: The fdisk command displaying the current partition list

Finally, you need to mount your partition as follows:
# mount /dev/sdb1 /mnt

Remove The Grub Password

a) If you are using Debian Linux you need to modify /boot/grub/menu.lst file:
# cd /mnt/boot/grub
# vi menu.lst

b) If you are using Suse Linux you need to modify /etc/grub.conf file
# cd /mnt/etc
# vi grub.conf

If you are using Red Hat / Fedora Linux you need to modify /boot/grub/grub.conf file
# cd /mnt/boot/grub
# vi grub.conf
Once file opened using a text editor, locate password line and remove it:
password --md5 $1$9sdflksdf/sdf44k

Fig.03: Removing The Grub Password Line

Save and close the file. Reboot the system.
# sync; sync
# cd /
# umount /mnt
# reboot

Change Root password

Now your Grub password is removed from the system. You need to login to the system without any password. To restore root user password boot system into a single user mode.

1) At Grub prompt press 'e' to edit command before booting.

2) Select second line

3) Again, press 'e' to edit selected command

4) Type the following line
init=/bin/sh
OR
init=/bin/bash

Fig.04: Booting into a single user mode using Grub

4) Press 'b' to boot system.

5) Now you are at shell prompt. Just change root password with the passwd command:
# passwd
Please note that depend upon your Linux distribution you may need to remount root partition in read/write mode before running the above command.

What is lost+found directory in Linux/Unix?

This is an important directory which is useful for recovering files which are not properly closed due to many reason such as power failure. Lost+Found is created by system at the time of Linux OS installation for each partition we create.

In other words we can say the mounted folder contains this lost+found folder. This folder contains the files with no links and files to be recovered. Any file to be recovered is kept in this folder. fsck command is used to recover these files.

This folder is not present in all the folders. Here is my HDD df command output.

surendra@linuxnix.com:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 37G 9.8G 26G 29%
/udev 3.9G 4.0K 3.9G 1%

/devtmpfs 1.6G 920K 1.6G 1% /runnone 5.0M 0 5.0M 0% /run/lock

none 3.9G 1.8M 3.9G 1% /run/shm

none 100M 44K 100M 1% /run/user

cgroup 3.9G 0 3.9G 0% /sys/fs/cgroup

/dev/sda5 37G 8.5G 27G 25% /home

/dev/sda7 499G 298G 177G 63% /opt

If I go in to / or /home or /opt folder I can see this folder.

surendra@linuxnix.com:~$ cd /homesurendra@linuxnix.com:/home$ pwd/home

surendra@linuxnix.com:/home$ ls

lost+found surendra

surendra@linuxnix.com:/home$

So in other words we can say for each partition this folder is present so that what ever files to be recovered with fsck can be kept in this folder. This folder is not accessible by normal user.

How to check/repair (fsck) filesystem after crash or power-outage

At some point your system will crash and you need to perform a manual repair of your file system. A typical situation would be power loss while you are working on the system. You reboot and the system stops and indicates you must perform a manual repair of the system using fsck.

fsck (file system consistency check) is a command used to check filesystem for consistency errors and repair them on Linux filesystems. This tool is important for maintaining data integrity so should be run regularly, especially after an unforeseen reboot (crash, power-outage).

Usage: fsck [-sACVRTNP] [-t fs-optlist] [filesystem] [fs-specific-options] 

Filesystem can be either a device's name (e.g. /dev/hda) or its mount point. fsck run with no options will check all devices in /etc/fstab. It might be neccesary to run fsck from single-user mode

Note: You need to be "root" to use any of the below mentioned command

* Take system down to runlevel one: # init 1

* Unmount file system, for example if it is /home (/dev/sda2) file system then type command:

# umount /home OR  # umount /dev/sda2

* Now run fsck on the partition: # fsck /dev/sda2

* Specify the file system type using -t option:  

# fsck -t ext3 /dev/sda2 OR  # fsck.ext3 /dev/sda2

fsck will check the file system and ask which problems should be fixed or corrected. If you don't wanna type y every time then you can use pass -y option to fsck: 

# fsck -y /dev/sda2

Please note if any files are recovered then they are placed in /home/lost+found directory by fsck command.

* Once fsck finished, remount the file system: # mount /home

Top 20 OpenSSH Server Best Security Practices

Default Config Files and SSH Port

§  /etc/ssh/sshd_config - OpenSSH server configuration file.

§  /etc/ssh/ssh_config - OpenSSH client configuration file.

§  ~/.ssh/ - Users ssh configuration directory.

§  ~/.ssh/authorized_keys or ~/.ssh/authorized_keys - Lists the public keys (RSA or DSA) that can be used to log into the user’s account

§  /etc/nologin - If this file exists, sshd refuses to let anyone except root log in.

§  /etc/hosts.allow and /etc/hosts.deny : Access controls lists that should be enforced by tcp-wrappers are defined here.

§  SSH default port : TCP 22

#1: Disable OpenSSH Server

Workstations and laptop can work without OpenSSH server. If you need not to provide the remote login and file transfer capabilities of SSH, disable and remove the SSHD server. CentOS / RHEL / Fedora Linux user can disable and remove openssh-server with yum command:

# chkconfig sshd off

# yum erase openssh-server

Debian / Ubuntu Linux user can disable and remove the same with apt-get command:

# apt-get remove openssh-server

You may need to update your iptables script to remove ssh exception rule. Under CentOS / RHEL / Fedora edit the files /etc/sysconfig/iptables and /etc/sysconfig/ip6tables. Once done restart iptables service:

# service iptables restart

# service ip6tables restart

#2: Only Use SSH Protocol 2

SSH protocol version 1 (SSH-1) has man-in-the-middle attacks problems and security vulnerabilities. SSH-1 is obsolete and should be avoided at all cost. Open sshd_config file and make sure the following line exists:

Protocol 2

#3: Limit Users' SSH Access

By default all systems user can login via SSH using their password or public key. Sometime you create UNIX / Linux user account for ftp or email purpose. However, those user can login to system using ssh. They will have full access to system tools including compilers and scripting languages such as Perl, Python which can open network ports and do many other fancy things. One of my client has really outdated php script and an attacker was able to create a new account on the system via a php script. However, attacker failed to get into box via ssh because it wasn't in AllowUsers.

Only allow root, vivek and jerry user to use the system via SSH, add the following to sshd_config:

AllowUsers root vivek jerry

Alternatively, you can allow all users to login via SSH but deny only a few users, with the following line:

DenyUsers saroj anjali foo

You can also configure Linux PAM allows or deny login via the sshd server. You can allow list of group name to access or deny access to the ssh.

#4: Configure Idle Log Out Timeout Interval

User can login to server via ssh and you can set an idel timeout interval to avoid unattended ssh session. Open sshd_config and make sure following values are configured:

ClientAliveInterval 300

ClientAliveCountMax 0

You are setting an idle timeout interval in seconds (300 secs = 5 minutes). After this interval has passed, the idle user will be automatically kicked out (read as logged out). See how to automatically log BASH / TCSH / SSH users out after a period of inactivity for more details.

#5: Disable .rhosts Files

Don't read the user's ~/.rhosts and ~/.shosts files. Update sshd_config with the following settings:

IgnoreRhosts yes

SSH can emulate the behavior of the obsolete rsh command, just disable insecure access via RSH.

#6: Disable Host-Based Authentication

To disable host-based authentication, update sshd_config with the following option:

HostbasedAuthentication no

#7: Disable root Login via SSH

There is no need to login as root via ssh over a network. Normal users can use su or sudo (recommended) to gain root level access. This also make sure you get full auditing information about who ran privileged commands on the system via sudo. To disable root login via SSH, update sshd_config with the following line:

PermitRootLogin no

However, bob made excellent point:

Saying "don't login as root" is h******t. It stems from the days when people sniffed the first packets of sessions so logging in as yourself and su-ing decreased the chance an attacker would see the root pw, and decreast the chance you got spoofed as to your telnet host target, You'd get your password spoofed but not root's pw. Gimme a break. this is 2005 - We have ssh, used properly it's secure. used improperly none of this 1989 will make a damn bit of difference. -Bob

#8: Enable a Warning Banner

Set a warning banner by updating sshd_config with the following line:

Banner /etc/issue

Sample /etc/issue file:

----------------------------------------------------------------------------------------------

You are accessing a XYZ Government (XYZG) Information System (IS) that is provided for authorized use only.

By using this IS (which includes any device attached to this IS), you consent to the following conditions:

+ The XYZG routinely intercepts and monitors communications on this IS for purposes including, but not limited to,

penetration testing, COMSEC monitoring, network operations and defense, personnel misconduct (PM),

law enforcement (LE), and counterintelligence (CI) investigations.

+ At any time, the XYZG may inspect and seize data stored on this IS.

+ Communications using, or data stored on, this IS are not private, are subject to routine monitoring,

interception, and search, and may be disclosed or used for any XYZG authorized purpose.

+ This IS includes security measures (e.g., authentication and access controls) to protect XYZG interests--not

for your personal benefit or privacy.

+ Notwithstanding the above, using this IS does not constitute consent to PM, LE or CI investigative searching

or monitoring of the content of privileged communications, or work product, related to personal representation

or services by attorneys, psychotherapists, or clergy, and their assistants. Such communications and work

product are private and confidential. See User Agreement for details.

----------------------------------------------------------------------------------------------

Above is standard sample, consult your legal team for exact user agreement and legal notice details.

#8: Firewall SSH Port # 22

You need to firewall ssh port # 22 by updating iptables or pf firewall configurations. Usually, OpenSSH server must only accept connections from your LAN or other remote WAN sites only.

Netfilter (Iptables) Configuration

Update /etc/sysconfig/iptables (Redhat and friends specific file) to accept connection only from 192.168.1.0/24 and 202.54.1.5/29, enter:

-A RH-Firewall-1-INPUT -s 192.168.1.0/24 -m state --state NEW -p tcp --dport 22 -j ACCEPT

-A RH-Firewall-1-INPUT -s 202.54.1.5/29 -m state --state NEW -p tcp --dport 22 -j ACCEPT

If you've dual stacked sshd with IPv6, edit /etc/sysconfig/ip6tables (Redhat and friends specific file), enter:

 -A RH-Firewall-1-INPUT -s ipv6network::/ipv6mask -m tcp -p tcp --dport 22 -j ACCEPT

Replace ipv6network::/ipv6mask with actual IPv6 ranges.

*BSD PF Firewall Configuration

If you are using PF firewall update /etc/pf.conf as follows:

pass in on $ext_if inet proto tcp from {192.168.1.0/24, 202.54.1.5/29} to $ssh_server_ip port ssh flags S/SA synproxy state

#9: Change SSH Port and Limit IP Binding

By default SSH listen to all available interfaces and IP address on the system. Limit ssh port binding and change ssh port (by default brute forcing scripts only try to connects to port # 22). To bind to 192.168.1.5 and 202.54.1.5 IPs and to port 300, add or correct the following line:

Port 300

ListenAddress 192.168.1.5

ListenAddress 202.54.1.5

A better approach to use proactive approaches scripts such as fail2ban or denyhosts (see below).

#10: Use Strong SSH Passwords and Passphrase

It cannot be stressed enough how important it is to use strong user passwords and passphrase for your keys. Brute force attack works because you use dictionary based passwords. You can force users to avoid passwords against a dictionary attack and use john the ripper tool to find out existing weak passwords. Here is a sample random password generator (put in your ~/.bashrc):

genpasswd() {

          local l=$1

          [ "$l" == "" ] && l=20

          tr -dc A-Za-z0-9_ < /dev/urandom | head -c ${l} | xargs

}

Run it:

genpasswd 16

Output:

uw8CnDVMwC6vOKgW

#11: Use Public Key Based Authentication

Use public/private key pair with password protection for the private key. See how to use RSA and DSA key based authentication. Never ever use passphrase free key (passphrase key less) login.

#12: Use Keychain Based Authentication

keychain is a special bash script designed to make key-based authentication incredibly convenient and flexible. It offers various security benefits over passphrase-free keys. See how to setup and use keychain software.

#13: Chroot SSHD (Lock Down Users To Their Home Directories)

By default users are allowed to browse the server directories such as /etc/, /bin and so on. You can protect ssh, using os based chroot or use special tools such as rssh. With the release of OpenSSH 4.8p1 or 4.9p1, you no longer have to rely on third-party hacks such as rssh or complicated chroot(1) setups to lock users to their home directories. See this blog post about new ChrootDirectory directive to lock down users to their home directories.

#14: Use TCP Wrappers

TCP Wrapper is a host-based Networking ACL system, used to filter network access to Internet. OpenSSH does supports TCP wrappers. Just update your /etc/hosts.allow file as follows to allow SSH only from 192.168.1.2 172.16.23.12 :

sshd : 192.168.1.2 172.16.23.12

See this FAQ about setting and using TCP wrappers under Linux / Mac OS X and UNIX like operating systems.

#15: Disable Empty Passwords

You need to explicitly disallow remote login from accounts with empty passwords, update sshd_config with the following line:

PermitEmptyPasswords no

Ubuntu / Debian Linux Regenerate OpenSSH Host Keys

Q. How do I regenerate OpenSSH sshd server host keys stored in /etc/ssh/ssh_host_* files? Can I safely regenerate ssh host keys using remote ssh session as my existing ssh connections shouldn't be interrupted?

A. To regenerate keys you need to delete old files and reconfigure openssh-server. It is also safe to run following commands over remote ssh session. Your existing session shouldn't be interrupted.

Step # 1: Delete old ssh host keys

Login as the root and type the following command:

# /bin/rm /etc/ssh/ssh_host_*

Step # 2: Reconfigure OpenSSH Server

Now create a new set of keys, enter:

# dpkg-reconfigure openssh-server

Sample output:

Creating SSH2 RSA key; this may take some time ...

Creating SSH2 DSA key; this may take some time ...

Restarting OpenBSD Secure Shell server: sshd.

Step # 3: Update all ssh client(s) known_hosts files

Finally, you need to update ~/.ssh/known_hosts files, otherwise everyone will see an error message:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

Someone could be eavesdropping on you right now (man-in-the-middle attack)!

It is also possible that the RSA host key has just been changed.

The fingerprint for the RSA key sent by the remote host is

f6:67:01:41:e6:20:06:4b:4b:fa:4b:c1:f1:45:45:e0.

Please contact your system administrator.

Add correct host key in /home/vivek/.ssh/known_hosts to get rid of this message.

Offending key in /home/vivek/.ssh/known_hosts:12

RSA host key for 202.54.xx.abc has changed and you have requested strict checking.

Host key verification failed.

= WARNING: UNPROTECTED PRIVATE KEY FILE SSH Related==================

ISSUE :

Failed to start SSH server : Starting sshd:@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: UNPROTECTED PRIVATE KEY FILE! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ Permissions 0775 for '/etc/ssh/ssh_host_key' are too open. It is recommended that your private key files are NOT accessible by others. This private key will be ignored. bad permissions: ignore key: /etc/ssh/ssh_host_key Could not load host key: /etc/ssh/ssh_host_key @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: UNPROTECTED PRIVATE KEY FILE! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ Permissions 0775 for '/etc/ssh/ssh_host_rsa_key' are too open. It is recommended that your private key files are NOT accessible by others. This private key will be ignored. bad permissions: ignore key: /etc/ssh/ssh_host_rsa_key Could not load host key: /etc/ssh/ssh_host_rsa_key @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: UNPROTECTED PRIVATE KEY FILE! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ Permissions 0775 for '/etc/ssh/ssh_host_dsa_key' are too open. It is recommended that your private key files are NOT accessible by others. This private key will be ignored. bad permissions: ignore key: /etc/ssh/ssh_host_dsa_key Could not load host key: /etc/ssh/ssh_host_dsa_key Disabling protocol version 1. Could not load host key Disabling protocol version 2. Could not load host key sshd: no hostkeys available -- exiting. [FAILED]

Solution :

chmod 600 /etc/ssh/ssh_host_rsa_key

chmod 600 /etc/ssh/ssh_host_dsa_key

this will set your key files to the proper setting once again and then you can...

service sshd stop

service sshd start

Issue :

To delete Key of SSH

> # vi ~/.ssh/know_host then delete