post

Create a containerized machine learning model

After data scientists have created a machine learning model, it has to be deployed into production. To run it on different infrastructures, using containers and exposing the model via a REST API is a common way to deploy a machine learning model. This article demonstrates how to roll out a TensorFlow machine learning model, with a REST API delivered by Connexion in a container with Podman.

Preparation

First, install Podman with the following command:

sudo dnf -y install podman

Next, create a new folder for the container and switch to that directory.

mkdir deployment_container && cd deployment_container

REST API for the TensorFlow model

The next step is to create the REST-API for the machine learning model. This github repository contains a pretrained model, and well as the setup already configured for getting the REST API working.

Clone this in the deployment_container directory with the command:

git clone https://github.com/svenboesiger/titanic_tf_ml_model.git

prediction.py & ml_model/

The prediction.py file allows for a Tensorflow prediction, while the weights for the 20x20x20 neural network are located in folder ml_model/.

swagger.yaml

The file swagger.yaml defines the API for the Connexion library using the Swagger specification. This file contains all of the information necessary to configure your server to provide input parameter validation, output response data validation, URL endpoint definition.

As a bonus Connexion will provide you also with a simple but useful single page web application that demonstrates using the API with JavaScript and updating the DOM with it.

swagger: "2.0" info: description: This is the swagger file that goes with our server code version: "1.0.0" title: Tensorflow Podman Article consumes: - "application/json" produces: - "application/json" basePath: "/" paths: /survival_probability: post: operationId: "prediction.post" tags: - "Prediction" summary: "The prediction data structure provided by the server application" description: "Retrieve the chance of surviving the titanic disaster" parameters: - in: body name: passenger required: true schema: $ref: '#/definitions/PredictionPost' responses: '201': description: 'Survival probability of an individual Titanic passenger' definitions: PredictionPost: type: object

server.py & requirements.txt

server.py  defines an entry point to start the Connexion server.

import connexion app = connexion.App(__name__, specification_dir='./') app.add_api('swagger.yaml') if __name__ == '__main__': app.run(debug=True)

requirements.txt defines the python requirements we need to run the program.

connexion tensorflow pandas

Containerize!

For Podman to be able to build an image, create a new file called “Dockerfile” in the deployment_container directory created in the preparation step above:

FROM fedora:28 # File Author / Maintainer MAINTAINER Sven Boesiger <donotspam@ujelang.com> # Update the sources RUN dnf -y update --refresh # Install additional dependencies RUN dnf -y install libstdc++ RUN dnf -y autoremove # Copy the application folder inside the container ADD /titanic_tf_ml_model /titanic_tf_ml_model # Get pip to download and install requirements: RUN pip3 install -r /titanic_tf_ml_model/requirements.txt # Expose ports EXPOSE 5000 # Set the default directory where CMD will execute WORKDIR /titanic_tf_ml_model # Set the default command to execute # when creating a new container CMD python3 server.py

Next, build the container image with the command:

podman build -t ml_deployment .

Run the container

With the Container image built and ready to go, you can run it locally with the command:

podman run -p 5000:5000 ml_deployment

Navigate to http://0.0.0.0:5000/ui in your web browser to access the Swagger/Connexion UI and to test-drive the model:

Of course you can now also access the model with your application via the REST-API.

post

Design faster web pages, part 3: Font and CSS tweaks

Welcome back to this series of articles on designing faster web pages. Part 1 and part 2 of this series covered how to lose browser fat through optimizing and replacing images. This part looks at how to lose additional fat in CSS (Cascading Style Sheets) and fonts.

Tweaking CSS

First things first: let’s look at where the problem originates. CSS was once a huge step forward. You can use it to style several pages from a central style sheet. Nowadays, many web developers use frameworks like Bootstrap.

While these frameworks are certainly helpful, many people simply copy and paste the whole framework. Bootstrap is huge; the “minimal” version of 4.0 is currently 144.9 KB. Perhaps in the era of terabytes of data, this isn’t much. But as they say, even small cattle makes a mess.

Look back at the getfedora.org example. Recall in part 1, the first analysis showed the CSS files used nearly ten times more space than the HTML itself. Here’s a display of the stylesheets used:

That’s nine different stylesheets. Many styles in them that are also unused on the page.

Remove, merge, and compress/minify

The font-awesome CSS inhabits the extreme end of included, unused styles. There are only three glyphs of the font used on the page. To make that up in KB, the font-awesome CSS used at getfedora.org is originally 25.2 KB. After cleaning out all unused styles, it’s only 1.3 KB. This is only about 4% of its original size! For Bootstrap CSS, the difference is 118.3 KB original, and 13.2 KB after removing unused styles.

The next question is, must there be a bootstrap.css and a font-awesome.css? Or can they be combined? Yes, they can. That doesn’t save much file space, but the browser now requests fewer files to succesfully render the page.

Finally, after merging the CSS files, try to remove unused styles and minify them. In this way, you save 10.1 KB for a final size of 4.3 KB.

Unfortunately, there’s no packaged “minifier” tool in Fedoras repositories yet. However, there are hundreds of online services to do that for you. Or you can use CSS-HTML-JS Minify, which is Python, and therefore easy to isntall. There’s not an available tool to purify CSS, but there are web services like UnCSS.

Font improvement

CSS3 came with something a lot of web developer like. They could define fonts the browser downloads in the background to render the page. Since then, a lot of web designers are very happy, especially after they discovered the usage of icon fonts for web design. Font sets like Font Awesome are quiet popular today and widely used. Here’s the size of that content:

current free version 912 glyphs/icons, smallest set ttf 30.9KB, woff 14.7KB, woff2 12.2KB, svg 107.2KB, eot 31.2

So the question is, do you need all the glyphs? In all probability, no. You can get rid of them with FontForge, but that’s a lot of work. You could also use Fontello. Use the public instance, or set up your own, as it’s free software and available on Github.

The downside of such customized font sets is you must host the font by yourself. You can’t use other online font services to provide updates. But this may not really be a downside, compared to faster performance.

Conclusion

Now you’ve done everything you can to the content itself, to minimize what the browser loads and interprets. From now on, only tricks with the administration of the server can help.

One easy to do, but which many people do wrong, is decide on some intelligent caching. For instance, a CSS or picture file can be cached for a week. Whatever you do, if you use a proxy service like Cloudflare or build your own proxy, minimze the pages first. Users like fast loading pages. They’ll (silently) thank you for it, and the server will have a smaller load, too.

post

4 cool new projects to try in COPR for October 2018

COPR is a collection of personal repositories for software that isn’t carried in the standard Fedora repositories. Some software doesn’t conform to standards that allow easy packaging. Or it may not meet other Fedora standards, despite being free and open source. COPR can offer these projects outside the standard set of Fedora Fedora packages. Software in COPR isn’t supported by Fedora infrastructure or signed by the project. However, it can be a neat way to try new or experimental software.

Here’s a set of new and interesting projects in COPR.

GitKraken

GitKraken is a useful git client for people who prefer a graphical interface over command-line, providing all the features you expect. Additionally, GitKraken can create repositories and files, and has a built-in editor. A useful feature of GitKraken is the ability to stage lines or hunks of files, and to switch between branches fast. However, in some cases, you may experience performance issues with larger projects.

Installation instructions

The repo currently provides GitKraken for Fedora 27, 28, 29 and Rawhide, and for OpenSUSE Tumbleweed. To install GitKraken, use these commands:

sudo dnf copr enable elken/gitkraken sudo dnf install gitkraken

Music On Console

Music On Console player, or mocp, is a simple console audio player. It has an interface similar to the Midnight Commander and is easy use. You simply navigate to a directory with music files and select a file or directory to play. In addition, mocp provides a set of commands, allowing it to be controlled directly from command line.

Installation instructions

The repo currently provides Music On Console player for Fedora 28 and 29. To install mocp, use these commands:

sudo dnf copr enable Krzystof/Moc sudo dnf install moc

cnping

Cnping is a small graphical ping tool for IPv4, useful for visualization of changes in round-trip time. It offers an option to control the time period between each packet as well as the size of data sent. In addition to the graph shown, cnping provides basic statistics on round-trip times and packet loss.

Installation instructions

The repo currently provides cnping for Fedora 27, 28, 29 and Rawhide. To install cnping, use these commands:

sudo dnf copr enable dreua/cnping sudo dnf install cnping

Pdfsandwich

Pdfsandwich is a tool for adding text to PDF files which contain text in an image form — such as scanned books. It uses optical character recognition (OCR) to create an additional layer with the recognized text behind the original page. This can be useful for copying and working with the text.

Installation instructions

The repo currently provides pdfsandwich for Fedora 27, 28, 29 and Rawhide, and for EPEL 7. To install pdfsandwich, use these commands:

sudo dnf copr enable merlinm/pdfsandwich sudo dnf install pdfsandwich
post

Design faster web pages, part 2: Image replacement

Welcome back to this series on building faster web pages. The last article talked about what you can achieve just through image compression. The example started with 1.2MB of browser fat, and reduced down to a weight of 488.9KB. That’s still not fast enough! This article continues the browser diet to lose more fat. You might think that partway through this process things are a bit crazy, but once finished, you’ll understand why.

Preparation

Once again this article starts with an analysis of the web pages. Use the built-in screenshot function of Firefox to make a screenshot of the entire page. You’ll also want to install Inkscape using sudo:

$ sudo dnf install inkscape

If you want to know how to use Inkscape, there are already several articles in Fedora Magazine. This article will only explain some basic tasks for optimizing an SVG for web use.

Analysis

Once again, this example uses the getfedora.org web page.

Getfedora page with graphics marked

Getfedora page with graphics marked

This analysis is better done graphically, which is why it starts with a screenshot. The screenshot above marks all graphical elements of the page. In two cases or better in four cases, the Fedora websites team already used measures to replace images. The icons for social media are glyphs from a font and the language selector is an SVG.

There are several options for replacing:

HTML5 Canvas

Briefly, HTML5 Canvas is an HTML element that allows you to draw with the help of scripts, mostly JavaScript, although it’s not widely used yet. As you draw with the help of scripts, the element can also be animated. Some examples of what you can achieve with HTML Canvas include this triangle pattern, animated wave, and text animation. In this case, though, it seems not to be the right choice.

CSS3

With Cascading Style Sheets you can draw shapes and even animate them. CSS is often used for drawing elements like buttons. However, more complicated graphics via CSS are usually only seen in technical demonstration pages. This is because graphics are still better done visually as with coding.

Fonts

The usage of fonts for styling web pages is another way, and Fontawesome is quiet popular. For instance, you could replace the Flavor and the Spin icons with a font in this example. There is a negative side to using this method, which will be covered in the next part of this series, but it can be done easily.

SVG

This graphics format has existed for a long time and was always supposed to be used in the browser. For a long time not all browsers supported it, but that’s history. So the best way to replace pictures in this example is with SVG.

Optimizing SVG for the web

To optimize an SVG for internet use requires several steps.

SVG is an XML dialect. Components like circle, rectangle, or text paths are described with nodes. Each node is an XML element. To keep the code clean, an SVG should use as few nodes as possible.

The SVG example is a circular icon with a coffee mug on it. You have 3 options to describe it with SVG.

Circle element with the mug on top

<circle style="opacity:1;fill:#717d82;fill-opacity:1;stroke:none;stroke-width:9.51950836;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;paint-order:markers fill stroke" id="path36" cx="68.414307" cy="130.71523" r="3.7620001" />

Circular path with the mug on top

<path style="opacity:1;fill:#717d82;fill-opacity:1;stroke:none;stroke-width:1.60968435;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;paint-order:markers fill stroke" d="m 68.414044,126.95318 a 3.7618673,3.7618673 0 0 0 -3.76153,3.76204 3.7618673,3.7618673 0 0 0 3.76153,3.76205 3.7618673,3.7618673 0 0 0 3.76206,-3.76205 3.7618673,3.7618673 0 0 0 -3.76206,-3.76204 z" id="path20" />

single path

<path style="opacity:1;fill:#717d82;fill-opacity:1;stroke:none;stroke-width:1.60968435;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;paint-order:markers fill stroke" d="m 68.414044,126.95318 a 3.7618673,3.7618673 0 0 0 -3.76153,3.76204 3.7618673,3.7618673 0 0 0 3.76153,3.76205 3.7618673,3.7618673 0 0 0 3.76206,-3.76205 3.7618673,3.7618673 0 0 0 -3.76206,-3.76204 z m -1.21542,0.92656 h 2.40554 c 0.0913,0.21025 0.18256,0.42071 0.27387,0.63097 h 0.47284 v 0.60099 h -0.17984 l -0.1664,1.05989 h 0.24961 l -0.34779,1.96267 -0.21238,-0.003 -0.22326,1.41955 h -2.12492 l -0.22429,-1.41955 -0.22479,0.003 -0.34829,-1.96267 h 0.26304 l -0.16692,-1.05989 h -0.1669 v -0.60099 h 0.44752 c 0.0913,-0.21026 0.18206,-0.42072 0.27336,-0.63097 z m 0.12608,0.19068 c -0.0614,0.14155 -0.12351,0.28323 -0.185,0.42478 h 2.52336 c -0.0614,-0.14155 -0.12248,-0.28323 -0.18397,-0.42478 z m -0.65524,0.63097 v 0.21911 l 0.0594,5.2e-4 h 3.35844 l 0.0724,-5.2e-4 v -0.21911 z m 0.16846,0.41083 0.1669,1.05937 h 2.80603 l 0.16693,-1.05937 -1.57046,0.008 z m -0.061,1.25057 0.27956,1.5782 1.34411,-0.0145 1.34567,0.0145 0.28059,-1.5782 z m 1.62367,1.75441 -1.08519,0.0124 0.19325,1.2299 h 1.79835 l 0.19328,-1.2299 z" id="path2714" inkscape:connector-curvature="0" />

You probably can see the code becomes more complex and needs more characters to describe it. More characters in a file result, of course, in a larger size.

Node cleaning

If you open an example SVG in Inkscape and press F2, that activates the Node tool. You should see something like this:

Inkscape - Node tool activated

Inkscape – Node tool activated

There are 5 nodes that aren’t necessary in this example — the ones in the middle of the lines. To remove them, select them one by one with the activated Node tool and press the Del key. After this, select the nodes which define this lines and make them corners again using the toolbar tool.

Inkscape - Node tool make node a corner

Inkscape – Node tool make node a corner

Without fixing the corners, handles are used that define the curve, which gets saved and will increase file size. You have to do this node cleaning by hand, as it can’t be effectively automated. Now you’re ready for the next stage.

Use the Save as function and choose Optimized svg. A dialogue window opens where you can select what to remove or keep.

Inkscape - Dialog window for save as optimized SVG

Inkscape – Dialog window for save as optimized SVG

Even the little SVG in this example got down from 3.2 KB to 920 bytes, less than a third of its original size.

Back to the getfedora page: The grey voronoi pattern used in the background of the main section, after our optimization from Part 1 of this series, is down to 164.1 KB versus the original 211.12 KB size.

The original SVG it was exported from is 1.9 MB in size. After these SVG optimization steps, it’s only 500.4KB. Too big? Well, the current blue background is 564.98 KB in size. But there’s only a small difference between the SVG and the PNG.

Compressed files

$ ls -lh insgesamt 928K -rw-r--r--. 1 user user 161K 19. Feb 19:44 grey-pattern.png -rw-rw-r--. 1 user user 160K 18. Feb 12:23 grey-pattern.png.gz -rw-r--r--. 1 user user 489K 19. Feb 19:43 greyscale-pattern-opti.svg -rw-rw-r--. 1 user user 112K 19. Feb 19:05 greyscale-pattern-opti.svg.gz

This is the output of a small test I did to visualize this topic. You should probably see that the raster graphic — the PNG — is already compressed and can’t be anymore. The opposite is the SVG, an XML file. This is just text and can compressed, to less then a fourth of its size. As a result it is now around 50 KB smaller in size than the PNG.

Modern browsers can handle compressed files natively. Therefore, a lot of web servers have switched on mod_deflate (Apache) and gzip (nginx). That’s how we save space during delivery. Check out if it’s enabled at your server here.

Tooling for production

First of all, nobody wants to always optimize SVG in Inkscape. You can run Inkscape without a GUI in batch mode, but there’s no option to convert from Inkscape SVG to optimized SVG. You can only export raster graphics this way. But there are alternatives:

  • SVGO (which seems not actively developed)
  • Scour

This example will use scour for optimization. To install it:

$ sudo dnf install scour

To automatically optimize an SVG file, run scour similarly to this:

[user@localhost ]$ scour INPUT.svg OUTPUT.svg -p 3 --create-groups --renderer-workaround --strip-xml-prolog --remove-descriptive-elements --enable-comment-stripping --disable-embed-rasters --no-line-breaks --enable-id-stripping --shorten-ids

This is the end of part two, in which you learned how to replace raster images with SVG and how to optimize it for usage. Stay tuned to the Fedora Magazine for part three, coming soon.

post

Design faster web pages, part 1: Image compression

Lots of web developers want to achieve fast loading web pages. As more page views come from mobile devices, making websites look better on smaller screens using responsive design is just one side of the coin. Browser Calories can make the difference in loading times, which satisfies not just the user but search engines that rank on loading speed. This article series covers how to slim down your web pages with tools Fedora offers.

Preparation

Before you sart to slim down your web pages, you need to identify the core issues. For this, you can use Browserdiet. It’s a browser add-on available for Firefox, Opera and Chrome and other browsers. It analyzes the performance values of the actual open web page, so you know where to start slimming down.

Next you’ll need some pages to work on. The example screenshot shows a test of getfedora.org. At first it looks very simple and responsive.

Browser Diet - values of getfedora.org

Browser Diet – values of getfedora.org

However, BrowserDiet’s page analysis shows there are 1.8MB in files downloaded. Therefore, there’s some work to do!

Web optimization

There are over 281 KB of JavaScript files, 203 KB more in CSS files, and 1.2 MB in images. Start with the biggest issue — the images. The tool set you need for this is GIMP, ImageMagick, and optipng. You can easily install them using the following command:

sudo dnf install gimp imagemagick optipng

For example, take the following file which is 6.4 KB:

First, use the file command to get some basic information about this image:

$ file cinnamon.png cinnamon.png: PNG image data, 60 x 60, 8-bit/color RGBA, non-interlaced 

The image — which is only in grey and white — is saved in 8-bit/color RGBA mode. That’s not as efficient as it could be.

Start GIMP so you can set a more appropriate color mode. Open cinnamon.png in GIMP. Then go to Image>Mode and set it to greyscale. Export the image as PNG with compression factor 9. All other settings in the export dialog should be the default.

$ file cinnamon.png cinnamon.png: PNG image data, 60 x 60, 8-bit gray+alpha, non-interlaced 

The output shows the file’s now in 8bit gray+alpha mode. The file size has shrunk from 6.4 KB to 2.8 KB. That’s already only 43.75% of the original size. But there’s more you can do!

You can also use the ImageMagick tool identify to provide more information about the image.

$ identify cinnamon2.png cinnamon.png PNG 60x60 60x60+0+0 8-bit Grayscale Gray 2831B 0.000u 0:00.000

This tells you the file is 2831 bytes. Jump back into GIMP, and export the file. In the export dialog disable the storing of the time stamp and the alpha channel color values to reduce this a little more. Now the file output shows:

$ identify cinnamon.png cinnamon.png PNG 60x60 60x60+0+0 8-bit Grayscale Gray 2798B 0.000u 0:00.000

Next, use optipng to losslessly optimize your PNG images. There are other tools that do similar things, including advdef (which is part of advancecomp), pngquant and pngcrush.

Run optipng on your file. Note that this will replace your original:

$ optipng -o7 cinnamon.png ** Processing: cinnamon.png 60x60 pixels, 2x8 bits/pixel, grayscale+alpha Reducing image to 8 bits/pixel, grayscale Input IDAT size = 2720 bytes Input file size = 2812 bytes Trying: zc = 9 zm = 8 zs = 0 f = 0 IDAT size = 1922 zc = 9 zm = 8 zs = 1 f = 0 IDAT size = 1920 Selecting parameters: zc = 9 zm = 8 zs = 1 f = 0 IDAT size = 1920 Output IDAT size = 1920 bytes (800 bytes decrease) Output file size = 2012 bytes (800 bytes = 28.45% decrease)

The option -o7 is the slowest to process, but provides the best end results. You’ve knocked 800 more bytes off the file size, which is now 2012 bytes.

To resize all of the PNGs in a directory, use this command:

$ optipng -o7 -dir=<directory> *.png

The option -dir lets you give a target directory for the output. If this option is not used, optipng would overwrite the original images.

Choosing the right file format

When it comes to pictures for the usage in the internet, you have the choice between:

JPG-LS and JPG 2000 are not widely used. Only a few digital cameras support these formats, so they can be ignored. aPNG is an animated PNG, and not widely used either.

You could save a few bytes more through changing the compression rate or choosing another file format. The first option you can’t do in GIMP, as it’s already using the highest compression rate. As there are no alpha channels in the picture, you can choose JPG as file format instead. For now use the default value of 90% quality — you could change it down to 85%, but then alias effects become visible. This saves a few bytes more:

$ identify cinnamon.jpg cinnamon.jpg JPEG 60x60 60x60+0+0 8-bit sRGB 2676B 0.000u 0:00.000 

Alone this conversion to the right color space and choosing JPG as file format brought down the file size from 23 KB to 12.3 KB, a reduction of nearly 50%.

PNG vs. JPG: quality and compression rate

So what about the rest of the images? This method would work for all the other pictures, except the Fedora “flavor” logos and the logos for the four foundations. Those are presented on a white background.

One of the main differences between PNG and JPG is that JPG has no alpha channel. Therefore it can’t handle transparency. If you rework these images by using a JPG on a white background, you can reduce the file size from 40.7 KB to 28.3 KB.

Now there are four more images you can rework: the backgrounds. For the grey background, set the mode to greyscale again. With this bigger picture, the savings also is bigger. It shrinks from 216.2 KB to 51.0 KB — it’s now barely 25% of its original size. All in all, you’ve shrunk 481.1 KB down to 191.5 KB — only 39.8% of the starting size.

Quality vs. Quantity

Another difference between PNG and JPG is the quality. PNG is a lossless compressed raster graphics format. But JPG loses size through compression, and thus affects quality. That doesn’t mean you shouldn’t use JPG, though. But you have to find a balance between file size and quality.

Achievement

This is the end of Part 1. After following the techniques described above, here are the results:

You brought image size down to 488.9 KB versus 1.2MB at the start. That’s only about a third of the size, just through optimizing with optipng. This page can probably be made to load faster still. On the scale from snail to hypersonic, it’s not reached racing car speed yet!

Finally you can check the results in Google Insights, for example:

In the Mobile area the page gathered 10 points on scoring, but is still in the Medium sector. It looks totally different for the Desktop, which has gone from 62/100 to 91/100 and went up to Good. As mentioned before, this test isn’t the be all and end all. Consider scores such as these to help you go in the right direction. Keep in mind you’re optimizing for the user experience, and not for a search engine.

post

Even better image upscaling with Waifu2x

You may have already seen the article here in the Magazine about upscaling bitmap images with better quality. That article covered a few utilities to achieve good results, but there’s always room for enhancement. Meet Waifu2x, a sophisticated tool that uses deep convolutional neural networks, or machine learning for short. Therefore it benefits from trained model data sets. Read on to see how well it performs to upscale your images, and how to get the same benefits even without an Nvidia card.

Here are some examples. First, a character that will be familiar to Fedora fans:

Making the Beefy Miracle hot dog four times larger keeps it nearly perfect!

Here’s another example:

Upscaling a drawing from an old book delivers really outstanding results.

Because Waifu2x performs the calculations using Nvidia CUDA computing platform, it requires an Nvidia graphics card. This article, however, covers a more portable version called Waifu2x-converter-cpp that runs everywhere.

Waifu2x-converter-cpp doesn’t require a dedicated Nvidia GPU. Instead, it leverages the OpenCV and OpenCL libraries. Although the overall performance is slower compared to the original code, it’s portable and more people are able to use it. And of course it runs on Fedora!

Follow the directions below to give it a spin.

Installing Waifu2x-converter-cpp in Fedora

To build the application from source, first install build dependencies:

sudo dnf install -y gcc-c++ cmake opencv-devel opencl-utils-devel

Then grab the source code, build it and install the files:

git clone https://github.com/DeadSix27/waifu2x-converter-cpp.git cd waifu2x-converter-cpp/build cmake -DLIB_SUFFIX=64 ../ make sudo make install

Tell the dynamic library linker to index the directory with locally installed libraries:

sudo ldconfig /usr/local/lib64

Now everything is ready for upscaling. The example command for processing an image is as follows:

waifu2x-converter-cpp --scale_ratio 2 -i /path/to/input_file -o /path/to/output_file

Don’t miss this very helpful command for extras, such as switching OpenCL on/off, playing with noise level, and so on:

waifu2x-converter-cpp --help

Using a graphical front-end for Waifu2x

Another step forward is to install Qtwaifu2x, a graphical tool that lets you escape Fedora terminal and process images with comfort and convenience.

Practically all you have to do is hit Browse and Start to launch the upscaling task.

Installing Qtwaifu2x in Fedora

First, install the main build dependency:

sudo dnf install -y qt5-qtbase-devel

Then get the code and build it:

git clone https://github.com/cmdrkotori/qtwaifu2x.git cd qtwaifu2x qmake-qt5 && make

To run the application from the current directory:

./qtwaifu2x

The main window has quite a few options, but it’s usually enough to provide the input file and hit the Start button.

Brew yourself a cup of tea and come back in a while to enjoy the perfectly upscaled image!

post

Understand Fedora memory usage with top

Have you used the top utility in a terminal to see memory usage on your Fedora system? If so, you might be surprised to see some of the numbers there. It might look like a lot more memory is consumed than your system has available. This article will explain a little more about memory usage, and how to read these numbers.

Memory usage in real terms

The way the operating system (OS) uses memory may not be self-evident. In fact, some ingenious, behind-the-scenes techniques are at play. They help your OS use memory more efficiently, without involving you.

Most applications are not self contained. Instead, each relies on sets of functions collected in libraries. These libraries are also installed on the system. In Fedora, the RPM packaging system ensures that when you install an app, any libraries on which it relies are installed, too.

When an app runs, the OS doesn’t necessarily load all the information it uses into real memory. Instead, it builds a map to the storage where that code is stored, called virtual memory. The OS then loads only the parts it needs. When it no longer needs portions of memory, it might release or swap them out as appropriate.

This means an app can map a very large amount of virtual memory, while using less real memory on the system at one time. It might even map more RAM than the system has available! In fact, across a whole OS that’s often the case.

In addition, related applications may rely on the same libraries. The Linux kernel in your Fedora system often shares memory between applications. It doesn’t need to load multiple copies of the same library for related apps. This works similarly for separate instances of the same app, too.

Without understanding these details, the output of the top application can be confusing. The following example will clarify this view into memory usage.

Viewing memory usage in top

If you haven’t tried yet, open a terminal and run the top command to see some output. Hit Shift+M to see the list sorted by memory usage. Your display may look slightly different than this example from a running Fedora Workstation:

There are three columns showing memory usage to examine: VIRT, RES, and SHR. The measurements are currently shown in kilobytes (KB).

The VIRT column is the virtual memory mapped for this process. Recall from the earlier description that virtual memory is not actual RAM consumed. For example, the GNOME Shell process gnome-shell is not actually consuming over 3.1 gigabytes of actual RAM. However, it’s built on a number of lower and higher level libraries. The system must map each of those to ensure they can be loaded when necessary.

The RES column shows you how much actual (resident) memory is consumed by the app. In the case of GNOME Shell, that’s about 180788 KB. The example system has roughly 7704 MB of physical memory, which is why the memory usage shows up as 2.3%.

However, of that number, at least 88212 KB is shared memory, shown in the SHR column. This memory might be, for example, library functions that other apps also use. This means the GNOME Shell is using about 92 MB on its own not shared with other processes. Notice that other apps in the example share an even higher percentage of their resident memory. In some apps, the shared portion is the vast majority of the memory usage.

There is a wrinkle here, which is that sometimes processes communicate with each other via memory. That memory is also shared, but can’t necessarily be detected by a utility like top. So yes — even the above clarifications still have some uncertainty!

A note about swap

Your system has another facility it uses to store information, which is swap. Typically this is an area of slower storage (like a hard disk). If the physical memory on the system fills up as needs increase, the OS looks for portions of memory that haven’t been needed in a while. It writes them out to the swap area, where they sit until needed later.

Therefore, prolonged, high swap usage usually means a system is suffering from too little memory for its demands. Sometimes an errant application may be at fault. Or, if you see this often on your system, consider upgrading your machine’s memory, or restricting what you run.


Photo courtesy of Stig Nygaard, via Flickr (CC BY 2.0).

post

4 tips for better tmux sessions

The tmux utility, a terminal multiplexer, lets you treat your terminal as a multi-paned window into your system. You can arrange the configuration, run different processes in each, and generally make better use of your screen. We introduced some readers to this powerful tool in this earlier article. Here are some tips that will help you get more out of tmux if you’re getting started.

This article assumes your current prefix key is Ctrl+b. If you’ve remapped that prefix, simply substitute your prefix in its place.

Set your terminal to automatically use tmux

One of the biggest benefits of tmux is being able to disconnect and reconnect to sesions at wilI. This makes remote login sessions more powerful. Have you ever lost a connection and wished you could get back the work you were doing on the remote system? With tmux this problem is solved.

However, you may sometimes find yourself doing work on a remote system, and realize you didn’t start a session. One way to avoid this is to have tmux start or attach every time you login to a system with in interactive shell.

Add this to your remote system’s ~/.bash_profile file:

if [ -z "$TMUX" ]; then tmux attach -t default || tmux new -s default fi

Then logout of the remote system, and log back in with SSH. You’ll find you’re in a tmux session named default. This session will be regenerated at next login if you exit it. But more importantly, if you detach from it as normal, your work is waiting for you next time you login — especially useful if your connection is interrupted.

Of course you can add this to your local system as well. Note that terminals inside most GUIs won’t use the default session automatically, because they aren’t login shells. While you can change that behavior, it may result in nesting that makes the session less usable, so proceed with caution.

Use zoom to focus on a single process

While the point of tmux is to offer multiple windows, panes, and processes in a single session, sometimes you need to focus. If you’re in a process and need more space, or to focus on a single task, the zoom command works well. It expands the current pane to take up the entire current window space.

Zoom can be useful in other situations too. For instance, imagine you’re using a terminal window in a graphical desktop. Panes can make it harder to copy and paste multiple lines from inside your tmux session. If you zoom the pane, you can do a clean copy/paste of multiple lines of data with ease.

To zoom into the current pane, hit Ctrl+b, z. When you’re finished with the zoom function, hit the same key combo to unzoom the pane.

Bind some useful commands

By default tmux has numerous commands available. But it’s helpful to have some of the more common operations bound to keys you can easily remember. Here are some examples you can add to your ~/.tmux.conf file to make sessions more enjoyable:

bind r source-file ~/.tmux.conf \; display "Reloaded config"

This command rereads the commands and bindings in your config file. Once you add this binding, exit any tmux sessions and then restart one. Now after you make any other future changes, simply run Ctrl+b, r and the changes will be part of your existing session.

bind V split-window -h bind H split-window

These commands make it easier to split the current window across a vertical axis (note that’s  Shift+V) or across a horizontal axis (Shift+H).

If you want to see how all keys are bound, use Ctrl+B, ? to see a list. You may see keys bound in copy-mode first, for when you’re working with copy and paste inside tmux. The prefix mode bindings are where you’ll see ones you’ve added above. Feel free to experiment with your own!

Use powerline for great justice

As reported in a previous Fedora Magazine article, the powerline utility is a fantastic addition to your shell. But it also has capabilities when used with tmux. Because tmux takes over the entire terminal space, the powerline window can provide more than just a better shell prompt.

Screenshot of tmux powerline in git folder

If you haven’t already, follow the instructions in the Magazine’s powerline article to install that utility. Then, install the addon using sudo:

sudo dnf install tmux-powerline

Now restart your session, and you’ll see a spiffy new status line at the bottom. Depending on the terminal width, the default status line now shows your current session ID, open windows, system information, date and time, and hostname. If you change directory into a git-controlled project, you’ll see the branch and color-coded status as well.

Of course, this status bar is highly configurable as well. Enjoy your new supercharged tmux session, and have fun experimenting with it.


Photo by Pamela Saunders on Unsplash.

post

5 cool music player apps

Do you like music? Then Fedora may have just what you’re looking for. This article introduces different music player apps that run on Fedora. You’re covered whether you have an extensive music library, a small one, or none at all. Here are four graphical application and one terminal-based music player that will have you jamming.

Quod Libet

Quod Libet is a complete manager for your large audio library. If you have an extensive audio library that you would like not just listen to, but also manage, Quod Libet might a be a good choice for you.

Quod Libet can import music from multiple locations on your disk, and allows you to edit tags of the audio files — so everything is under your control. As a bonus, there are various plugins available for anything from a simple equalizer to a last.fm sync. You can also search and play music directly from Soundcloud.

Quod Libet works great on HiDPI screens, and is available as an RPM in Fedora or on Flathub in case you run Silverblue. Install it using Gnome Software or the command line:

$ sudo dnf install quodlibet

Audacious

If you like a simple music player that could even look like the legendary Winamp, Audacious might be a good choice for you.

Audacious probably won’t manage all your music at once, but it works great if you like to organize your music as files. You can also export and import playlists without reorganizing the music files themselves.

As a bonus, you can make it look likeWinamp. To make it look the same as on the screenshot above, go to Settings / Appearance, select Winamp Classic Interface at the top, and choose the Refugee skin right below. And Bob’s your uncle!

Audacious is available as an RPM in Fedora, and can be installed using the Gnome Software app or the following command on the terminal:

$ sudo dnf install audacious

Lollypop

Lollypop is a music player that provides great integration with GNOME. If you enjoy how GNOME looks, and would like a music player that’s nicely integrated, Lollypop could be for you.

Apart from nice visual integration with the GNOME Shell, it woks nicely on HiDPI screens, and supports a dark theme.

As a bonus, Lollypop has an integrated cover art downloader, and a so-called Party Mode (the note button at the top-right corner) that selects and plays music automatically for you. It also integrates with online services such as last.fm or libre.fm.

Available as both an RPM in Fedora or a Flathub for your Silverblue workstation, install it using the Gnome Software app or using the terminal:

$ sudo dnf install lollypop

Gradio

What if you don’t own any music, but still like to listen to it? Or you just simply love radio? Then Gradio is here for you.

Gradio is a simple radio player that allows you to search and play internet radio stations. You can find them by country, language, or simply using search. As a bonus, it’s visually integrated into GNOME Shell, works great with HiDPI screens, and has an option for a dark theme.

Gradio is available on Flathub which works with both Fedora Workstation and Silverblue. Install it using the Gnome Software app.

sox

Do you like using the terminal instead, and listening to some music while you work? You don’t have to leave the terminal thanks to sox.

sox is a very simple, terminal-based music player. All you need to do is to run a command such as:

$ play file.mp3

…and sox will play it for you. Apart from individual audio files, sox also supports playlists in the m3u format.

As a bonus, because sox is a terminal-based application, you can run it over ssh. Do you have a home server with speakers attached to it? Or do you want to play music from a different computer? Try using it together with tmux, so you can keep listening even when the session closes.

sox is available in Fedora as an RPM. Install it by running:

$ sudo dnf install sox

Photo by Malte Wingen on Unsplash.

post

Decentralize common Fedora apps with Cjdns

Are you worried about a few huge corporations controlling the web? Don’t like censorship on centralized social media sites like facebook and twitter? You need to decentralize! The internet was designed to be decentralized. Many common activities, from social media to email to voice calls, don’t actually require a centralized service.

The basic requirement for any peer to peer application is that the peers be able to reach each other. This is impossible today for most people using IP4 behind NAT (as with most household routers). The IP4 address space was exhausted over a decade ago. Most people are in “IP4 NAT Jail.”

Your device is assigned a private IP, and translated to the public IP by the router. Without port forwarding to a specific private IP, incoming TCP connections or UDP sessions can’t tell where to forward to, and are dropped. As a result, nothing can connect to your device. You must connect to various public servers to do anything. IP4 NAT Jail forces centralization.

The simplest solution to this problem is IPv6. However, most US consumer internet providers do not offer usable IPv6. For instance, if the IPv6 prefix changes every few days, the devices are not addressable except via a dynamic DNS server. Furthermore, on a mobile device like a laptop, most WiFi does not offer IPv6 either. So you can’t use Mobile IP6 to have a stable address.

You can work around this using a VPN like OpenVPN (included in Fedora) to a centralized server with a public IP4 — perhaps one you provide yourself by renting a Virtual Personal Server. But then packets to and from your device have to travel to and from the VPN server first. You can also use a tunnel broker like he.net.

If you and your peers already have stable IPv6 addresses, you can use these for the sample applications to be showcased. But most people need to use something else.

DNS is also essentially a centrally controlled service, so this article’s two sample applications avoid the use of DNS.  Email and SIP applications have built-in address books that work just as well.  Think of your stable IPv6 address as a “phone number.”

IPv6 Overlay Mesh VPN with Cjdns

The Cjdns package (included in Fedora) implements a global IPv6 mesh by connecting to several peers instead of a centralized server. Each node has a public/private key pair. The IPv6 is the truncated SHA512 hash of the public key, preventing spoofing.

  • Packets are end to end encrypted — relays can be untrusted.
  • Packets are source routed, allowing seamless upgrades of and experimentation with routing algorithms.  (This is safe thanks to anti-spoofing.)
  • The data for routing comes from a Distributed Hash Table listing the peers of each node.
  • Peers can be explicitly configured as UDP tunnels, or auto-configured on ethernet via layer 2 protocol 0xfc00.

With Cjdns installed, you have a stable, “unspoofable” (standard cryptographic caveats apply) IPv6 address that can be used with any IPv6 ready application. Your recipient must also use the Cjdns protocol, but this isn’t much of an obstacle since it’s easier to install Cjdns than convince US ISPs to provide usable IPv6.

Install Cjdns

To install and enable the Cjdns service persistently, run these commands:

$ sudo dnf install cjdns cjdns-tools cjdns-selinux $ sudo systemctl enable --now cjdns
$ peerStats 18:03:14:56:c2:1e v20.0000.0000.0000.0019.681v1s7k3af1q2cf09txpw309zdf4q0mn7mtq0wr544dz98stwr0.k ESTABLISHED in 6kb/s out 15kb/s LOS 8 "outer"

This generates a /etc/cjdroute.conf file, pre-populated with random keys and passwords. If there’s already a Cjdns node on your LAN as above, you’re done. But more likely, there was no output from peerStats. In that case you now need to configure one or more UDP tunnels. First, you must discover the random UDP port used.

$ sudo grep bind /etc/cjdroute.conf // Port to bind the admin RPC server to. "bind": "127.0.0.1:11234", "bind": "0.0.0.0:26041", "bind": "[::]:26041", // Alternatively bind to just one device "bind": "all",

In this example, the random UDP port is 26041 for both IPv6 and IPv4. Your port will be different. Allow incoming sessions for this port.

$ sudo firewall-cmd --zone=public --add-port=26041/udp success $ sudo firewall-cmd --runtime-to-permanent success

Now you need to edit the config to add a peer. Hopefully, you are somewhat familiar with configs using JSON syntax. You must add an entry for a UDP peer using your favorite text editor, such as vim. Here is one provided on a VPS. Search for IPv4, and add the indicated stanza after connectTo, inside the braces:

$ sudo vim /etc/cjdroute.conf "168.235.90.18:26041": { "login": "fedora", "password":"zvkxv604fqx0zn9trhw5hjxwp3u4v2u", "publicKey":"lhj54c2xnczfurpw42d0h1bvc4qquclb4dw72q50tc83ucmm9zt0.k", "peerName":"nyc.gathman.org" },

For the changes to take effect, restart cjdns.

$ sudo systemctl restart cjdns $ peerStats 168.235.90.18:26041 v20.0000.0000.0000.0017.lhj54c2xnczfurpw42d0h1bvc4qquclb4dw72q50tc83ucmm9zt0.k ESTABLISHED in 0kb/s out 0kb/s "nyc.gathman.org" $ ping h.sea.gathman.org PING h.sea.gathman.org(h.sea.gathman.org (fceb:7fc0:c62c:9cd9:2971:e3ff:aee2:6e08)) 56 data bytes 64 bytes from h.sea.gathman.org (fceb:7fc0:c62c:9cd9:2971:e3ff:aee2:6e08): icmp_seq=1 ttl=42 time=87.6 ms

You can now ping any node in the global IPv6 mesh. CAUTION: All those nodes can now directly connect to your device. The default Fedora firewall will block all incoming connections be default — but be careful what you allow in. Be sure to consult the package README for additional security notes.

The fedora password to this nyc VPS may not be up indefinitely, so you need some more peers. Consult a list of public peers or peer with your Fedora friends.

Decentralize Email applications

You can decentralize almost any email client included in Fedora that supports IPv6, such as alpine or Thunderbird. This example uses mailx, a bare bones CLI mail client designed for teletypes. This makes configuration and use easy to show.

Similarly, you can use any of the MTAs supplied with Fedora, but this example uses opensmtpd, as it is simple, small, and secure. By default, opensmtpd stores incoming email in /var/spool/mail, which is perfect for personal decentralized use. You can, of course, use any mail store and client you prefer.

$ sudo dnf install mailx opensmtpd $ cat >~/.mailrc <<EOF set from="mylogin@[IPv6:fc02:fefe:dead:beef:cafe:babe:1234:5678] (Real Name)" set smtp=localhost EOF

Of course, you need to use your own local login, IPv6 and name.

To receive email, you will need to edit the opensmtpd config in /etc/opensmtpd/smtpd.conf. Here is a sample. (Note this article may wrap some of the “preformatted” lines, so use your head):

# This is the smtpd server system-wide configuration file. # See smtpd.conf(5) for more information. # To accept external mail, replace with: listen on all listen on fc02:fefe:dead:beef:cafe:babe:1234:5678 hostname "[IPv6:fc02:fefe:dead:beef:cafe:babe:1234:5678]" listen on localhost # If you edit the file, you have to run "smtpctl update table aliases" table aliases file:/etc/aliases # Uncomment the following to accept external mail for domain "example.org" #accept from any for domain "example.org" alias deliver to mbox accept from any for domain "[IPv6:fc02:fefe:dead:beef:cafe:babe:1234:5678]" alias deliver to mbox accept for local alias deliver to mbox accept for any relay hostname "[IPv6:fc02:fefe:dead:beef:cafe:babe:1234:5678]" 

Use your actual Cjdns IP, of course. When the opensmtpd config is ready, start it so you can receive emails. If your recipient is offline, opensmtpd stores your letter and retries periodically.

$ sudo systemctl enable --now opensmtpd $ sudo firewall-cmd --zone=public --add-service=smtp $ sudo firewall-cmd --runtime-to-permanent

Now send the author a dex (decentralized) email:

$ mailx -s "Fedora Article" \ "stuart@[IPv6:fcbc:b27:be6f:94dd:4225:792:c988:8ace]" <<EOF > Great article! > EOF

That sends an email to the author’s nyc vps — so don’t be surprised if you get a reply!

Alpine is a full featured console email client. After you install and run it the first time, you can decentralize it by editing ~/.pinerc and changing these basic config items:

# Sets domain part of From: and local addresses in outgoing. user-domain=[IPv6:fc02:fefe:dead:beef:cafe:babe:1234:5678] # List of SMTP servers for sending mail. smtp-server=localhost

Decentralize SIP applications

Linphone call screen

Linphone call screen

Using Cjdns for your voice calls gives you privacy and authentication. You can use any sip client that supports IP6. This example uses the linphone app included in Fedora.

$ sudo dnf install linphone $ sudo firewall-cmd --zone=public --add-service=sip --add-port=7078/udp --add-port=9078/udp $ sudo firewall-cmd --runtime-to-permanent
Linphone network config screen

Linphone network config screen

Run linphone on your desktop, and skip the account wizard. You don’t need logins and accounts with peer to peer. Select Options, Preferences and select Use IPv6 instead of IPv4 and Direct connection to internet. Enter your Cjdns IPv6 in Public IP address. Now select Options, Quit to completely exit linphone.

The version in Fedora doesn’t provide a way to configure your peer to peer contact, so you need to edit the config file. Find the [sip] section and change guess_hostname and contact:

$ vim ~/.linphonerc guess_hostname=0 contact="Real Name" <sip:mylogin@[fc02:fefe:dead:beef:cafe:babe:1234:5678]>

Now start linphone again, and add a Fedora friend with Cjdns to the addressbook using the same address syntax. Try a text message first, then give them a call.

Of course, there are many potential issues with audio and video in a VoIP app, which are not covered here. Usually, however, linphone just works. If you don’t have any friends, you can reach out to the author via dex email at the nyc node above.