Unix system administration commonly consists of repeating common and similar operations on different sets of targets. The notion of using, and stringing together, building block primitives such as cat, sort, strings, wc, sed and awk is a core tenet of Unix philosophy. //

By incorporating the data into the script itself, one can create powerful system administration tools, in the form of simple shell scripts, that consist merely of a single file.

The basic organization of such a script is a set of one or more data items which are commented out, as they are not actual commands, but commented in such a way that they can be distinguished from normal comments:

01: #!/bin/sh
02: #
03: # Here is the data set, and perhaps we will add some other comments here
04: #
05: ##DATA var1 var2 var3
06: ##DATA var1 var2 var3
07: ##DATA var1 var2 var3

As you can see, normal comments are commented out with one # character, but data items are commented with ##. Not only does this allow us the ability to parse through the script and easily identify which lines are data lines (as opposed to normal comments), but it also allows us to quickly disable a data line that we temporarily do not wish to use. Simply remove one of the # characters from the data line that is not to be used - it will not be parsed because it has no leading ##, but it still starts with #, and thus does not affect the script as it is still commented out.

The body of the script consists of a variable defining the path of the script itself (obviously the script needs to know the path to itself if it is to parse itself), and then a while loop that reads in every line of the script, but filters out (using grep) only those lines that are data lines, which begin with ##DATA :

08: myself=/usr/local/bin/script.sh
09:
10: while read line
11: do
12:
13:    if echo $line | grep "^##DATA"
14:    then
15:
16:      var1=`echo $line | awk '{print $2}'`
17:      var2=`echo $line | awk '{print $3}'`
18:      var3=`echo $line | awk '{print $4}'`
19:
20:      diff $var1 $var2 >> $var3
21:
22:    fi
23:
24: done < $myself

What is happening here is that the script, in line 08, defines the path to itself, and uses a "while read line" construct, the end of which in line 24 takes as input the name of the script itself. ...

cat is surely the best way to do this. Why use python when there is a program already written in C for this purpose? However, you might want to consider using xargs in case the command line length exceeds ARG_MAX and you need more than one cat. Using GNU tools, this is equivalent to what you already have:

find . -maxdepth 1 -type f -name 'input_file*' -print0 |
  sort -z |
  xargs -0 cat -- >>out

what is the fastest method to CREATE a thousands of files? Also, does it really matter if there is data in them, if they are just being deleted?

Using rsync is surprising fast and simple.

mkdir empty_dir
rsync -rd --delete empty_dir/    yourdirectory/

his text is a brief description of the features that are present in the Bash shell (version 5.2, 19 September 2022). The Bash home page is http://www.gnu.org/software/bash/.

This is Edition 5.2, last updated 19 September 2022, of The GNU Bash Reference Manual, for Bash, Version 5.2.

Bash contains features that appear in other popular shells, and some features that only appear in Bash. Some of the shells that Bash has borrowed concepts from are the Bourne Shell (sh), the Korn Shell (ksh), and the C-shell (csh and its successor, tcsh). The following menu breaks the features up into categories, noting which features were inspired by other shells and which are specific to Bash.

This manual is meant as a brief introduction to features found in Bash. The Bash manual page should be used as the definitive reference on shell behavior.

The bash manual states:

case word in [ [(] pattern [ | pattern ] ... ) list ;; ] ... esac

Each pattern examined is expanded using tilde expansion, parameter and variable expansion, arithmetic substitution, command substitution, and process substitution.

These few lines are not intended as a full-fledged debugging tutorial, but as hints and comments about debugging a Bash script.

set -x or set -o xtrace expands variables and prints a little + sign before the line.

set -v or set -o verbose does not expand the variables before printing.

Use set +x and set +v to turn off the above settings.

On the first line of the script, one can put #!/bin/sh -x (or -v) to have the same effect as set -x (or -v) later in the script.

Debug bash scripts:

So the basic example is to set PS4 to some plain text, e.g.:

 PS4="# "; set -x

or

 #!/bin/bash -x
 PS4="# "

a table that helps in quick comprehension of the topic.

Get started with Bash Shell script learning with practical examples. Also test your learning with practice exercises.

When you run a command in bash it will remember the location of that executable so it doesn't have to search the PATH again each time. So if you run the executable, then change the location, bash will still try to use the old location. You should be able to confirm this with hash -t pip3 which will show the old location.

If you run hash -d pip3 it will tell bash to forget the old location and should find the new one next time you try. //

for most bash features it's easier to use help instead of man, so here help hash

:(){ :|:& };:

The command shown in the heading is known as a Bash “Fork Bomb.”

A fork bomb is a denial-of-service attack where a process continuously creates child processes at an exponential rate, consuming system resources like CPU, memory, and process slots, ultimately causing the system to crash. //

To set limits for the current bash session:

Run ulimit -u to check the maximum number of processes you can have (e.g., 30593).
Run ulimit -u NUM, where NUM is significantly lower than your maximum (e.g., 1024).

Setting persistent user limits
The above method works unless the user reopens their terminal and runs the fork bomb again.

To set persistent user limits, add the same ulimit command to your ~/.bashrc or ~/.bash_profile file.

ulimit -u 1024 # Example for my system 

Setting persistent user limits
Configuring system-wide limits is similar to setting user limits, but involves editing a different file that manages system-wide process rules.

Typically, you would run sudo nano /etc/security/limits.conf and add the following user limits:
username hard nproc 1024
Remember to replace “username” with the user you wish to limit.

# let days=$(date --date=yesterday +%s)/86400 "even = days % 2" && [ $even -gt 0 ] && echo $days
19967

# let days=$(date --date=today +%s)/86400 "even = days % 2" && [ $even -gt 0 ] && echo $days || echo NOT
NOT

Bash let is a built-in command in Linux systems used for evaluating arithmetic expressions. Unlike other arithmetic evaluation and expansion commands, let is a simple command with its own environment. The let command also allows for arithmetic expansion.

Those variables are shell variables. To expand them as parameters to another program (ie expr), you need to use the $ prefix:

 expr $x / $y

The reason it complained is because it thought you were trying to operate on alphabetic characters (ie non-integer)

If you are using the Bash shell, you can achieve the same result using expression syntax:

echo $((x / y))

Or:

z=$((x / y))
echo $z

Why not use let; I find it much easier. Here's an example you may find useful:

start=`date +%s`
# ... do something that takes a while ...
sleep 71

end=`date +%s`
let deltatime=end-start
let hours=deltatime/3600
let minutes=(deltatime/60)%60
let seconds=deltatime%60
printf "Time spent: %d:%02d:%02d\n" $hours $minutes $seconds

Another simple example - calculate number of days since 1970:

 let days=$(date +%s)/86400

Infinite for Loops

You can also use this format of for loop to create an infinite loop. All you need do is remove all of the elements from the loop header, like this. This is "infinite.sh."

#!/bin/bash
for (( ; ; ))
do
 echo "Press Ctrl+C to stop..."
 sleep 1
done

You'll need to hit Ctrl+C to stop the loop.

 # loop through the indices and print elements
 for position in ${!bd_bands[@]};
 do
 echo "band ${bd_bands[$position]} is at index: $position"
 done

In the third part of the Bash Beginner Series, you'll learn to pass arguments to a bash shell script. You'll also learn about special bash shell variables.

An A-Z Index of the Linux command line: bash + utilities.