02-06-2019, 08:34 PM
And, Ampersand, and & in Linux
<div style="margin: 5px 5% 10px 5%;"><img src="http://www.sickgaming.net/blog/wp-content/uploads/2019/02/and-ampersand-and-in-linux.png" width="1135" height="617" title="" alt="" /></div><div><div><img src="http://www.sickgaming.net/blog/wp-content/uploads/2019/02/and-ampersand-and-in-linux.png" class="ff-og-image-inserted" /></div>
<p>Take a look at the <a href="https://www.linux.com/blog/learn/2019/1/linux-tools-meaning-dot">three</a> <a href="https://www.linux.com/blog/learn/2019/1/understanding-angle-brackets-bash">previous</a> <a href="https://www.linux.com/blog/learn/2019/1/more-about-angle-brackets-bash">articles</a>, and you will see that understanding the glue that joins them together is as important as recognizing the tools themselves. Indeed, tools tend to be simple and understanding what <i>mkdir</i>, <i>touch</i>, and <i>find</i> do (make a new directory, update a file, and find a file in the directory tree, respectively) in isolation from each other is easy.</p>
<p>But understanding what</p>
<pre>
mkdir test_dir 2>/dev/null || touch images.txt && find . -iname "*jpg" > backup/dir/images.txt &
</pre>
<p>does, and why we would write a command line like that is a whole different story.</p>
<p>It pays to look more closely at the sign and symbols that live between the commands. It will not only help you better understand how things work, but will also make you more proficient in chaining commands together to create compound instructions that will help you work more efficiently.</p>
<p>In this article and the next, we’ll be looking at the the ampersand (<code>&</code>) and its close friend, the pipe (<code>|</code>), and see how they can mean different things in different contexts.</p>
<h3>Behind the Scenes</h3>
<p>Let’s start simple and see how you can use <code>&</code> as a way of pushing a command to the background. The instruction:</p>
<pre>
cp -R original/dir/ backup/dir/
</pre>
<p>Copies all the files and subdirectories in <i>original/dir/</i> into <i>backup/dir/</i>. So far so simple. But if that turns out to be a lot of data, it could tie up your terminal for hours.</p>
<p>However, using:</p>
<pre>
cp -R original/dir/ backup/dir/ &
</pre>
<p>pushes the process to the background courtesy of the final <code>&</code>. This frees you to continue working on the same terminal or even to close the terminal and still let the process finish up. Do note, however, that if the process is asked to print stuff out to the standard output (like in the case of <code>echo</code> or <code>ls</code>), it will continue to do so, even though it is being executed in the background.</p>
<p>When you push a process into the background, Bash will print out a number. This number is the PID or the <i>Process’ ID</i>. Every process running on your Linux system has a unique process ID and you can use this ID to pause, resume, and terminate the process it refers to. This will become useful later.</p>
<p>In the meantime, there are a few tools you can use to manage your processes as long as you remain in the terminal from which you launched them:</p>
<ul>
<li>
<p><code>jobs</code> shows you the processes running in your current terminal, whether be it in the background or foreground. It also shows you a number associated with each job (different from the PID) that you can use to refer to each process:</p>
<pre>
$ <b>jobs</b> [1]- Running cp -i -R original/dir/* backup/dir/ & [2]+ Running find . -iname "*jpg" > backup/dir/images.txt &
</pre>
</li>
<li>
<p><code>fg</code> brings a job from the background to the foreground so you can interact with it. You tell <code>fg</code> which process you want to bring to the foreground with a percentage symbol (<code>%</code>) followed by the number associated with the job that <code>jobs</code> gave you:</p>
<pre>
$ <b>fg %1</b> # brings the cp job to the foreground
cp -i -R original/dir/* backup/dir/
</pre>
<p>If the job was stopped (see below), <code>fg</code> will start it again.</p>
</li>
<li>
<p>You can stop a job in the foreground by holding down [Ctrl] and pressing [Z]. This doesn’t abort the action, it pauses it. When you start it again with (<code>fg</code> or <code>bg</code>) it will continue from where it left off…</p>
<p>…Except for <a href="https://ss64.com/bash/sleep.html"><code>sleep</code></a>: the time a <code>sleep</code> job is paused still counts once <code>sleep</code> is resumed. This is because <code>sleep</code> takes note of the clock time when it was started, not how long it was running. This means that if you run <code>sleep 30</code> and pause it for more than 30 seconds, once you resume, <code>sleep</code> will exit immediately.</p>
</li>
<li>The <code>bg</code> command pushes a job to the background and resumes it again if it was paused:
<pre>
$ <b>bg %1</b>
[1]+ cp -i -R original/dir/* backup/dir/ &
</pre>
</li>
</ul>
<p>As mentioned above, you won’t be able to use any of these commands if you close the terminal from which you launched the process or if you change to another terminal, even though the process will still continue working.</p>
<p>To manage background processes from another terminal you need another set of tools. For example, you can tell a process to stop from a a different terminal with the <a href="https://bash.cyberciti.biz/guide/Sending_signal_to_Processes"><code>kill</code></a> command:</p>
<pre>
kill -s STOP <<i>PID</i>>
</pre>
<p>And you know the PID because that is the number Bash gave you when you started the process with <code>&</code>, remember? Oh! You didn’t write it down? No problem. You can get the PID of any running process with the <code>ps</code> (short for <i>processes</i>) command. So, using</p>
<pre>
ps | grep cp
</pre>
<p>will show you all the processes containing the string “<i>cp</i>“, including the copying job we are using for our example. It will also show you the PID:</p>
<pre>
$ <b>ps | grep cp</b>
14444 pts/3 00:00:13 cp
</pre>
<p>In this case, the PID is <i>14444</i>. and it means you can stop the background copying with:</p>
<pre>
kill -s STOP 14444
</pre>
<p>Note that <code>STOP</code> here does the same thing as [Ctrl] + [Z] above, that is, it pauses the execution of the process.</p>
<p>To start the paused process again, you can use the <code>CONT</code> signal:</p>
<pre>
kill -s CONT 14444
</pre>
<p>There is a good list of many of <a href="https://www.computerhope.com/unix/signals.htm">the main signals you can send a process here</a>. According to that, if you wanted to terminate the process, not just pause it, you could do this:</p>
<pre>
kill -s TERM 14444
</pre>
<p>If the process refuses to exit, you can force it with:</p>
<pre>
kill -s KILL 14444
</pre>
<p>This is a bit dangerous, but very useful if a process has gone crazy and is eating up all your resources.</p>
<p>In any case, if you are not sure you have the correct PID, add the <code>x</code> option to <code>ps</code>:</p>
<pre>
$ <b>ps x| grep cp</b>
14444 pts/3 D 0:14 cp -i -R original/dir/Hols_2014.mp4 original/dir/Hols_2015.mp4 original/dir/Hols_2016.mp4 original/dir/Hols_2017.mp4 original/dir/Hols_2018.mp4 backup/dir/ </pre>
<p>And you should be able to see what process you need.</p>
<p>Finally, there is nifty tool that combines <code>ps</code> and <code>grep</code> all into one:</p>
<pre>
$ <b>pgrep cp</b>
8 18 19 26 33 40 47 54 61 72 88 96 136 339 6680 13735 14444
</pre>
<p>Lists all the PIDs of processes that contain the string “<i>cp</i>“.</p>
<p>In this case, it isn’t very helpful, but this…</p>
<pre>
$ <b>pgrep -lx cp</b>
14444 cp
</pre>
<p>… is much better.</p>
<p>In this case, <code>-l</code> tells <code>pgrep</code> to show you the name of the process and <code>-x</code> tells <code>pgrep</code> you want an exact match for the name of the command. If you want even more details, try <code>pgrep -ax <i>command</i></code>.</p>
<h3>Next time</h3>
<p>Putting an <code>&</code> at the end of commands has helped us explain the rather useful concept of processes working in the background and foreground and how to manage them.</p>
<p>One last thing before we leave: processes running in the background are what are known as <i>daemons</i> in UNIX/Linux parlance. So, if you had heard the term before and wondered what they were, there you go.</p>
<p>As usual, there are more ways to use the ampersand within a command line, many of which have nothing to do with pushing processes into the background. To see what those uses are, we’ll be back next week with more on the matter.</p>
</div>
<div style="margin: 5px 5% 10px 5%;"><img src="http://www.sickgaming.net/blog/wp-content/uploads/2019/02/and-ampersand-and-in-linux.png" width="1135" height="617" title="" alt="" /></div><div><div><img src="http://www.sickgaming.net/blog/wp-content/uploads/2019/02/and-ampersand-and-in-linux.png" class="ff-og-image-inserted" /></div>
<p>Take a look at the <a href="https://www.linux.com/blog/learn/2019/1/linux-tools-meaning-dot">three</a> <a href="https://www.linux.com/blog/learn/2019/1/understanding-angle-brackets-bash">previous</a> <a href="https://www.linux.com/blog/learn/2019/1/more-about-angle-brackets-bash">articles</a>, and you will see that understanding the glue that joins them together is as important as recognizing the tools themselves. Indeed, tools tend to be simple and understanding what <i>mkdir</i>, <i>touch</i>, and <i>find</i> do (make a new directory, update a file, and find a file in the directory tree, respectively) in isolation from each other is easy.</p>
<p>But understanding what</p>
<pre>
mkdir test_dir 2>/dev/null || touch images.txt && find . -iname "*jpg" > backup/dir/images.txt &
</pre>
<p>does, and why we would write a command line like that is a whole different story.</p>
<p>It pays to look more closely at the sign and symbols that live between the commands. It will not only help you better understand how things work, but will also make you more proficient in chaining commands together to create compound instructions that will help you work more efficiently.</p>
<p>In this article and the next, we’ll be looking at the the ampersand (<code>&</code>) and its close friend, the pipe (<code>|</code>), and see how they can mean different things in different contexts.</p>
<h3>Behind the Scenes</h3>
<p>Let’s start simple and see how you can use <code>&</code> as a way of pushing a command to the background. The instruction:</p>
<pre>
cp -R original/dir/ backup/dir/
</pre>
<p>Copies all the files and subdirectories in <i>original/dir/</i> into <i>backup/dir/</i>. So far so simple. But if that turns out to be a lot of data, it could tie up your terminal for hours.</p>
<p>However, using:</p>
<pre>
cp -R original/dir/ backup/dir/ &
</pre>
<p>pushes the process to the background courtesy of the final <code>&</code>. This frees you to continue working on the same terminal or even to close the terminal and still let the process finish up. Do note, however, that if the process is asked to print stuff out to the standard output (like in the case of <code>echo</code> or <code>ls</code>), it will continue to do so, even though it is being executed in the background.</p>
<p>When you push a process into the background, Bash will print out a number. This number is the PID or the <i>Process’ ID</i>. Every process running on your Linux system has a unique process ID and you can use this ID to pause, resume, and terminate the process it refers to. This will become useful later.</p>
<p>In the meantime, there are a few tools you can use to manage your processes as long as you remain in the terminal from which you launched them:</p>
<ul>
<li>
<p><code>jobs</code> shows you the processes running in your current terminal, whether be it in the background or foreground. It also shows you a number associated with each job (different from the PID) that you can use to refer to each process:</p>
<pre>
$ <b>jobs</b> [1]- Running cp -i -R original/dir/* backup/dir/ & [2]+ Running find . -iname "*jpg" > backup/dir/images.txt &
</pre>
</li>
<li>
<p><code>fg</code> brings a job from the background to the foreground so you can interact with it. You tell <code>fg</code> which process you want to bring to the foreground with a percentage symbol (<code>%</code>) followed by the number associated with the job that <code>jobs</code> gave you:</p>
<pre>
$ <b>fg %1</b> # brings the cp job to the foreground
cp -i -R original/dir/* backup/dir/
</pre>
<p>If the job was stopped (see below), <code>fg</code> will start it again.</p>
</li>
<li>
<p>You can stop a job in the foreground by holding down [Ctrl] and pressing [Z]. This doesn’t abort the action, it pauses it. When you start it again with (<code>fg</code> or <code>bg</code>) it will continue from where it left off…</p>
<p>…Except for <a href="https://ss64.com/bash/sleep.html"><code>sleep</code></a>: the time a <code>sleep</code> job is paused still counts once <code>sleep</code> is resumed. This is because <code>sleep</code> takes note of the clock time when it was started, not how long it was running. This means that if you run <code>sleep 30</code> and pause it for more than 30 seconds, once you resume, <code>sleep</code> will exit immediately.</p>
</li>
<li>The <code>bg</code> command pushes a job to the background and resumes it again if it was paused:
<pre>
$ <b>bg %1</b>
[1]+ cp -i -R original/dir/* backup/dir/ &
</pre>
</li>
</ul>
<p>As mentioned above, you won’t be able to use any of these commands if you close the terminal from which you launched the process or if you change to another terminal, even though the process will still continue working.</p>
<p>To manage background processes from another terminal you need another set of tools. For example, you can tell a process to stop from a a different terminal with the <a href="https://bash.cyberciti.biz/guide/Sending_signal_to_Processes"><code>kill</code></a> command:</p>
<pre>
kill -s STOP <<i>PID</i>>
</pre>
<p>And you know the PID because that is the number Bash gave you when you started the process with <code>&</code>, remember? Oh! You didn’t write it down? No problem. You can get the PID of any running process with the <code>ps</code> (short for <i>processes</i>) command. So, using</p>
<pre>
ps | grep cp
</pre>
<p>will show you all the processes containing the string “<i>cp</i>“, including the copying job we are using for our example. It will also show you the PID:</p>
<pre>
$ <b>ps | grep cp</b>
14444 pts/3 00:00:13 cp
</pre>
<p>In this case, the PID is <i>14444</i>. and it means you can stop the background copying with:</p>
<pre>
kill -s STOP 14444
</pre>
<p>Note that <code>STOP</code> here does the same thing as [Ctrl] + [Z] above, that is, it pauses the execution of the process.</p>
<p>To start the paused process again, you can use the <code>CONT</code> signal:</p>
<pre>
kill -s CONT 14444
</pre>
<p>There is a good list of many of <a href="https://www.computerhope.com/unix/signals.htm">the main signals you can send a process here</a>. According to that, if you wanted to terminate the process, not just pause it, you could do this:</p>
<pre>
kill -s TERM 14444
</pre>
<p>If the process refuses to exit, you can force it with:</p>
<pre>
kill -s KILL 14444
</pre>
<p>This is a bit dangerous, but very useful if a process has gone crazy and is eating up all your resources.</p>
<p>In any case, if you are not sure you have the correct PID, add the <code>x</code> option to <code>ps</code>:</p>
<pre>
$ <b>ps x| grep cp</b>
14444 pts/3 D 0:14 cp -i -R original/dir/Hols_2014.mp4 original/dir/Hols_2015.mp4 original/dir/Hols_2016.mp4 original/dir/Hols_2017.mp4 original/dir/Hols_2018.mp4 backup/dir/ </pre>
<p>And you should be able to see what process you need.</p>
<p>Finally, there is nifty tool that combines <code>ps</code> and <code>grep</code> all into one:</p>
<pre>
$ <b>pgrep cp</b>
8 18 19 26 33 40 47 54 61 72 88 96 136 339 6680 13735 14444
</pre>
<p>Lists all the PIDs of processes that contain the string “<i>cp</i>“.</p>
<p>In this case, it isn’t very helpful, but this…</p>
<pre>
$ <b>pgrep -lx cp</b>
14444 cp
</pre>
<p>… is much better.</p>
<p>In this case, <code>-l</code> tells <code>pgrep</code> to show you the name of the process and <code>-x</code> tells <code>pgrep</code> you want an exact match for the name of the command. If you want even more details, try <code>pgrep -ax <i>command</i></code>.</p>
<h3>Next time</h3>
<p>Putting an <code>&</code> at the end of commands has helped us explain the rather useful concept of processes working in the background and foreground and how to manage them.</p>
<p>One last thing before we leave: processes running in the background are what are known as <i>daemons</i> in UNIX/Linux parlance. So, if you had heard the term before and wondered what they were, there you go.</p>
<p>As usual, there are more ways to use the ampersand within a command line, many of which have nothing to do with pushing processes into the background. To see what those uses are, we’ll be back next week with more on the matter.</p>
</div>