Category Archives: Computer – Linux

Server connectivity issues have been resolved

Over the past few months, there have been a variety of issues plaguing my server(s) and network.  These problems have come and gone, making arriving at a solution very difficult.

I have a fairly unique networking setup, utilizing a main router flashed with an open source firmware. This router is wireless bridged to another similar router, extending the wireless umbrella’s range.  The bridged router only handles wireless duties, and operates as a print server.  All of the servers are directly connected to the main router.

Initially, it seemed as there were dns issues, that I thought were entirely Frontier’s fault.  They seemed that they were, a few service calls were placed, and they eventually found a bad wire out at the pole.  Things seemed to get a bit better, but they were still tweaky.  

I tried a variety of things: reflashed and reconfigured the router, replaced the router with the backup router, even replaced the power brick.  The router was found to be running under extreme load, somewhere around 1.5 or so, with 4096 Active IP connections (100% — maxed out).

So, I rolled my sleeves up, started by enabling the built in firewall in the router, moving the main server out of the DMZ, and port forwarded required ports to it.  This all allowed me the ability to actually start checking the router’s configuration, but it would still periodically overheat and reboot on its own.

So, then, I was able to manage to get ssh enabled in the router, and connected to it via putty ssh client, so as to ease the webgui load…  Using the netstat command, I noticed an absolute flood of connections from one of the ancillary servers.

I tracked it down to a bit of malware that somehow got on one of the servers.  (No idea how, as I don’t surf the internet from there, or collect email.) All I can imagine is that on one of the rare occasions that I ran a web browser there, I clicked on an infected link.  What was going on was a piece of malware called cron64/tsm.  It sets up house in .bashtemp and .X19-unix directories, so as to pass casual scrutiny.  So, I cleaned everything, and the router is behaving much nicer now, as the router has been online for just about 48 hours WITHOUT a spontaneous reboot. 😉

The router’s load average is running around .15 now, occasionally going up to about .45.  The Active IP connections have dropped from the maxed out 4096 to around 200.  Things are back to working as they should be.

Now, what is cron64/tsm?  It’s a bitcoin miner.  They are fairly easy to get infected with, so I should take this time to spread the word….

BitCoins and crypto currency – these are the talk of the town. In December 2017, the value of a single BitCoin crossed $19,000 to nearly touch $20,000. It had been steadily increasing and analysts are comparing the Bitcoin run to the dot-com bubble of the late 1990s. This has induced cyber criminals to find out ways to surreptitiously steal/mine Bitcoins, and miner viruses are one such tool.

Bitcoins are created as a reward for a process known as mining. Bitcoin is a cryptocurrency and used as a worldwide payment system. The anonymity associated with the cryptocurrency has led to it being used for criminal, fraudulent and illegal activities. It is widely used in dark web transactions, drug trade, etc…,

Though predominantly used for nefarious activities, Bitcoins are being accepted by certain businesses and organizations for products, and services and can also be exchanged for other currencies, though to a limited extent.

BitCoin mining is a record-keeping service that is done using computer processing power. Bitcoin transactions are recorded in blockchains, which functions as a public ledger. The consistentency and completeness of the blockchain is maintained in an unalterable state by miners, who repeatedly verify and collect newly broadcast transactions – this is called as a block.  Cyber criminals infect vulnerable computers with BitCoin Miner Virus to steal computer processing power. This drastically affects the performance of the system. Hence, users must be able to detect and remove BitCoin Miner virus.

Symptoms of BitCoin Miner Virus:
Overusage of CPU and GPU
Overheating of system
Drastic slowing down of system
Sustained mining could break your PCs hardware.

How is the BitCoin Miner Malware Spread
The BitCoin miner malware are spread through numerous methods, such as email attachments, and embedded in compromised websites. This malware is also found embedded in Trojan Horse viruses. It has also been spread across Windows networks by exploiting the EternalBlue vulnerability.

How to Detect BitCoin Miner Virus
It is very difficult to detect the BitCoin Miner Virus as it is a fileless malware. Traditional antivirus solutions, and most modern virus protection software are NOT capable of detecting fileless malware. You may attempt manual removal, however, it requires considerable technical skill as you have to interact with registry entries such as ActiveScriptEventConsumer, EventFilter, IntervalTimerInstruction, AbsoluteTimerInstruction, and FilterToConsumerBinding. This is quite complicated and sensitive, and incorrect handling could brick your system.

Antivirus and Virus Protection Software
An efficient Antivirus solution that can detect and block fileless malware is what you need. Traditional solutions detect malware based on virus definitions, and hence they cannot detect fileless malware. The Comodo Antivirus which is a part of Comodo Internet Security and Comodo Advanced Protection solutions provides protection against all types of malware including fileless malware such as BitCoin Miner Virus. Comodo Antivirus provides cloud-based resilient default deny protection to block all unknown files including zero-day malware. All unknown files are automatically contained in a sophisticated virtual container where they are allowed to execute and their behavior is observed. This container combines a virtualization of COM interfaces, Disk, Registry, and Memory. The unknown file believes that it is making changes to the real environment, however, it is making changes only to the virtual system. The behavior of the file helps decide if the file is good or malicious.

This virtual containerization, which is unique to Comodo, is the only antivirus solution that can detect and remove the BitCoin Miner Virus.

The moral of this story is that if you aren’t using Comodo Internet Security, you should be.  It’s available for Windows, Mac and Linux….  And, it’s free.

The importance of staying with linux built-in package management…

I just upgraded my Fedora server from 20 to 22.  (I know that I have procrastinated..)  A couple issues were encountered.  The first of which was fairly catastrophic.  After the usual fedup (FEDora UPgrade, which by the way was rather short-lived, from FC18 until FC21, being replaced along with yum by ‘dnc’) stuff, upon reboot, the server hung when it was trying to load the Gnome Desktop Manager.  I really think that was probably a problem with Gnome 3 and my hardware (since I could log in fine using ssh), so I got around this by the following:

# sudo dnf install lightdm lightdm-gtk

# sudo sustemctl disable gdm

# sudo systemctl enable lightdm

(If you haven’t set up sudo, you really should)  After those steps, a simple reboot, and presto!  Finished booting just fine.

The next problem encountered was one that fully illustrates the importance of staying within a modern Linux distribution’s built-in package management system….

Modern Linux Distributions use package managers, a partial list follows:

RPM: Used by RHEL, Fedora, CentOS, Yellow Dog, OpenSUSE….

DEB: Used by Debian and Ubuntu, Mint, SteamOS, Trisquel….

(Those were the ‘big two’, encompassing many different distributions, but there are many more….

Most people know that my server handles an icecast/ices streaming server.  However, ices 0.4 refused to start, giving the following error:

/usr/local/icecast/bin/ices: error while loading shared libraries: libperl.so.5.18: cannot open shared object file: No such file or directory

As part of the upgrade, perl was upgraded from perl 5.18 to 5.20, and libperl.so now points to libperl.so.5.20.  The problem arose from the fact that ices 2 is the version in the package repository.  However, it has no mp3 support, instead streams ogg.  So, for that reason, I had compiled by hand ices 0.4.

So, the only way to fix this libperl issue was to re-compile ices. (A good reason to keep the src of programs that you compile in your /home dir).

So, I jumped to the ices0.4 subdirectory and did the following:

# ./configure    (so that ices would pick up the new perl library)

# make

# sudo make install

After that, everything is fine, but, as I’ve always said, in a modern Linux distribution, try to minimize hand compiled programs.  Not always possible, but definitely preferred.

😎

prune (shear) command

Well, I successfully managed to get the one line command I wrote to successfully delete all security camera entries older than 100 days.  The directory had grown to about 300GB, encompassing 14 months, so it was time to weed things out a bit.

The command I wrote into an executable file in my personal path was:  find $1 -mtime +100 -exec rm -R {} \;

I saved it in my path and called it prune.  The syntax it was to expect was:  prune <directory to prune with trailing />  I was then to remove all files and directories older than 100 days.

However, it didn’t work.  I scratched my head for awhile, discovered it worked if I manually typed in the find string.  Sometimes I can be such a dumb ass.  I didn’t realize that there already was a prune command in /usr/bin/ for some kind of graphing functions.  So, I renamed MY prune command to shear, and everything works as expected.

Oh, the 300GB of security camera video was reduced to 60GB.  My next step will be to put it in a cron, so that it’s automatic….  Too  bad that I’ll never get that 45 minutes back, LOL.

CRAB server firewall.iptables now fixed, so it will restart without a reboot.

For those of you familiar with my Linux server, y’all know that I run many custom BASH, Perl, and Python scripts.  We are going to discuss today my iptables firewall script, or actually the fact that I took the time to repair it.   Since upgrading the server to Fedora 20 in September 2014, my firewall script would start fine, but would choke on restarting, reloading, or running it over itself.  The end result would be that the server would sever all ports with the outside, effectively taking it down, needing a reboot to get it back online.  This was a real PITA, since I couldn’t really make any changes to the firewall without reboot.  I thought it was an end result of the OS changing from SysV init scripts to a systemd system.  How wrong I was….  The cause was simply a typo that I made while adding a feature, and a ‘done’ without a ‘while’ and a ‘fi’ without an ‘if’ stopped the script from accepting the restart and reload argument.  Most of the iptables rulesets were created before this typo, so it gave the ‘appearance’ that it was working properly.  Debugging a script of this size is a fairly daunting task, as the main firewall script is a healthy 50 pages, and its configuration file (one of a half dozen helper files) is about 7 pages long.  Since I’ve had a few questions about this firewall, I’ll share it with everybody…..

Download links are only viewable to logged in users.  All the other helper files, as well as these are all available upon request.  If you are a logged in user, and downloading these files, remove the .txt extension from the config and main files….

This is the screen information if the VERBOSE variable is set to 1 (Select the pop-out gadget to see this properly formatted)

Loader
Loading...

EAD Logo
Taking too long?

Reload Reload document

|

Open Open in new tab

This is the configuration file that allows for easy firewall config.     (Select the pop-out gadget to see this properly formatted)

Loader
Loading...

EAD Logo
Taking too long?

Reload Reload document

|

Open Open in new tab

This is the MAIN iptables firewall script that should be run at start.  (Select the pop-out gadget to see this properly formatted)

Loader
Loading...

EAD Logo
Taking too long?

Reload Reload document

|

Open Open in new tab

All in all, it was a fairly productive week off.  I have a few other minor syntax errors to chase down, but the firewall is operating within normal parameters…..