Porteus - Portable Linux

[HOWTO] Understanding Module Creation and Manipulation: Two Case Studies

To view or contibute to the discussion about this tutorial, visit this thread on our forum.

If you're like me, then you learn and understand concepts better as you use and apply them, rather than simply reading about them. In this tutorial, we'll get our hands dirty with some real examples of Porteus modules, by creating them, extracting them, editing them, and generally messing around with them. Much of this will be with command line tools, which offer the best way to really get "under the hood", but you can just as easily perform many of these steps from the GUI, using the right-click menus in your file manager. Please read the doc on our main site, "What are Porteus Modules" before proceeding.

Note: this tutorial was written for the 32-bit edition of Porteus (V1.0). If you are using the 64-bit edition, the same commands apply, but you should use 64-bit packages, or the modules won't work. Pick any package you like, but if it requires dependencies, your program won't run in our first example.

Conventions: As you read through this document, perform any actions inside codeblocks in a console. For example:

cd ~/Downloads

means you should enter the text, exactly as it appears, in your console. Do not execute text that is preceded by a # sign, these are comments to help describe the commands you are being asked to perform.

Case Number 1:

The first thing we're going to do is download a program that does not require any dependencies, and convert it into a module, then extract it to a folder, tinker with the files, and then convert that folder back to a module.

Ok, we're going to start off with obtaining a slackware package (rpms or debs would also suffice and can be converted to Porteus modules with deb2xzm or rpm2xzm, but since Porteus is slackware based, let's start with a slackware package):

Navigate your browser to:

http://pkgs.org/slackware-13.37/slacky-i486/aterm-1.0.1-i486-6sl.txz.html

scroll down on that page a bit, and you'll see a list of files and directories:

/install/
/install/slack-desc
/install/slack-required
/usr/
/usr/bin/
/usr/bin/aterm
/usr/doc/
/usr/doc/aterm-1.0.1/
/usr/doc/aterm-1.0.1/ChangeLog
/usr/doc/aterm-1.0.1/INSTALL
/usr/doc/aterm-1.0.1/README.configure
/usr/doc/aterm-1.0.1/aterm.SlackBuild
/usr/doc/aterm-1.0.1/slack-desc
/usr/man/
/usr/man/man1/
/usr/man/man1/aterm.1.gz

These are all of the files that are contained in the aterm slackware package (slackware packages have the .txz file extension). All of these files will be incorporated into our module in the same directory structure, with the exception of the files in /install/. The files in /install/ are used by the install or conversion scripts to create symlinks and other files that are not contained in the slackware package, but should be included in our module.

Now, scroll back up and click on "select mirror", then "binary package" to download this file to your Desktop. I'm logged in as root (you should log in as root as well, or use su for the command line apps), and I've saved it to: /root/Downloads/aterm-1.0.1-i486-6sl.txz

Open up a terminal. The first thing we're going to do is set up a clean workspace, to keep from cluttering up our system:

mkdir -p ~/temp/modwork/
#creates nested folders inside our home directory (i.e., /root/ or /home/guest/)

Now, let's move our slack package to our workspace and convert it into a module:

mv ~/Downloads/ ~/temp/modwork/
cd ~/temp/modwork/
txz2xzm aterm-1.0.1-i486-6sl.txz aterm-1.0.1-i486-6sl.xzm

Use tab autocompletion for the package name (i.e., enter mv ~/Downloads/aterm, then press TAB and the rest of the filename will autocomplete), and the module name can be anything you want, but it's easier to keep your modules oganized if you use a specific name that includes the version number.

Now, you have an .xzm file next to your .txz file in ~/temp/modwork/. This module is ready to use, as it has no dependencies. You can't activate a module from within your live filesystem --if you try to activate it, the system will relocate it to /mnt/live/memory/modules and then activate it (and it will disappear from your 'modwork' directory). Let's copy this to some place permanent instead:

cp aterm-1.0.1-i486-6sl.xzm /mnt/sdXN/porteus/optional
#where sdXN is the device and partition on which porteus is installed

Now, let's activate that module:

activate /mnt/sdXN/porteus/optional/aterm-1.0.1-i486-6sl.xzm

And start it up:

aterm

Now, you have another console program installed in Porteus, which can be activated by double clicking the module in /mnt/sdXN/porteus/optional, or you can move it to /mnt/sdXN/porteus/modules, and it will be automatically loaded into your system every time you boot up.

Let's say you wanted to see which files are inside your module. This can be accomplished in a few ways:

First, using 'lsxzm':

lsxzm aterm-1.0.1-i486-6sl.xzm
#this will list all of the files in the module

Also, you can use 'mloop':

mloop aterm-1.0.1-i486-6sl.xzm

this will mount the module on a loop, and the files will be available inside the /mnt/loop/ directory. You can then copy some or all of them to another location to edit them and compress them into a new module if you choose.

Finally, if the module is active, you can view the files in read-only format at /mnt/live/memory/images/aterm-1.0.1-i486-6sl.xzm/

Poke around using one of these methods, and you will see that the module contains the same files as the slackware package that we used to create the module (with the exception of the files in the packages /install directory, and the addition of symlinks, etc., created by the scripts in /install).

Now, perhaps you want to add or modify a file inside the module. You would need to extract the module, add or modify the file, and then compress it again:

cd ~/temp/modwork/
#we're going back to the module in our clean workspace to mess around with it
#because if we extract a module on a FAT or NTFS file system
#i.e. your /porteus/optional folder, if your drive is FAT or NTFS,
#all of the symlinks and file permissions will be lost.
mkdir aterm
xzm2dir aterm-1.0.1-i486-6sl.xzm aterm
#this extracts the xzm module to the 'aterm' directory you just created
mkdir -p aterm/usr/share/applications
cp /usr/share/applications/psinfo.desktop aterm/usr/share/applications/aterm.desktop
mcedit aterm/usr/share/applications/aterm.desktop
#copy an existing a desktop entry file and then add your text to create a menu entry for aterm
#change the Exec= line to Exec=aterm which is the file that will be executed
#change the name to aterm and change the description
dir2xzm aterm aterm-1.0.1-i486-modified.xzm
#compresses the files inside the aterm directory into a new module
# Now after activation you should have a menu entry for aterm which you can click on

By now, you should understand the basics of how to create convert, mount, extract and modify a module. If not, keep working with it. You can create a module out of any directory, so you can just create a handful of files inside a directory and start making modules, extracting them, etc.

Case No. 2

To learn a few more useful tools, let's take a look at a package that depends on another package which is not included with Porteus by default, and look at how that module can be constructed and manipulated:

This time, we're going to use slackyd, a command line package manager to obtain the slackware packages.

First, update slackyd:

slackyd -u

Now, we'll download the package without installing it (using the -g switch):

slackyd -g xpdf

Press y to download the package.

Now, let's go to where slackyd stores downloaded packages: /var/slackyd

cd /var/slackyd
#now, move the file to your workspace
mv xpdf-3.02pl6-i486-4.txz ~/temp/modwork/

We want to find out what dependencies this package has, so we're going to change directories again, install xpdf and then use slackyd to find the dependencies:

cd ~/temp/modwork/
installpkg xpdf-3.02pl6-i486-4.txz
#installs the package to the live filesystem; this install will not survive reboot unless you are
#saving your changes with the changes=cheatcode.
slackyd -d

The -d switch searches your existing system for missing dependencies. This will take a while, and will return required packages that are needed for xpdf, as well as any other packages. This is a portion of the output from that command:

Found 4 missing dependencies:
libruby.so.1.8 required by:
--> Package: kdebindings-3.5.12-i486-1 (/usr/bin/qtrubyinit)
--> Package: kdebindings-3.5.12-i486-1 (/usr/bin/krubyinit)
--> Package: koffice-3.5.12-i486-1 (/usr/lib/kde3/krossruby.so)
libnotify.so.1 required by:
--> Package: mozilla-firefox-4.0.1-i486-1 (/usr/lib/firefox-4.0.1/components/libmozgnome.so)
libQtCore.so.4 required by:
--> Package: wpa_supplicant-0.7.3-i486-3 (/usr/bin/wpa_gui)
libXm.so.2 required by:
--> Package: xpdf-3.02pl6-i486-4 (/usr/bin/xpdf)

You can see that libXm.so.2 is required by xpdf, and all of the other libraries are required by other programs in Porteus. Slackyd will prompt you to download packages for all of the missing dependencies it finds, so press Y to download lesstif (of which libXm is a part), but say N to decline downloading all of the others.

Now, we'll move lesstif to our workspace, install it and check again for missing dependencies:

mv /var/slackyd/lesstif-0.95.2-i486-1.txz ~/temp/modwork/
cd ~/temp/modwork/
#just in case you left it!
installpkg lesstif-0.95.2-i486-1.txz
slackyd -d

This time through, neither xpdf nor lesstif have any required dependencies, so we should have everything we need.

Since both packages are installed in our live system, let's check if xpdf works:

xpdf

And, indeed, it should.

If you're saving your changes, you should probably remove xpdf and lesstif in order to keep things tidy:

removepkg xpdf-3.02pl6-i486-4
removepkg lesstif-0.95.2-i486-1

Note that installpkg and removepkg are slackware tools for installing and removing slackware txz packages.

Now, you could use txz2xzm to convert each of these packages into a module, and then use Porteus-Module-Tools to merge those modules together. However, it's easier to install both packages to a fakeroot directory, and then turn that directory into a module:

#we're still in ~/temp/modwork/, cd here if you've left
mkdir xpdf
installpkg -root ~/temp/modwork/xpdf *.txz
#the -root switch specifies the fake root directory for the install, and *.txz applies the command to any file in the current directory that ends in .txz
#now, let's check to see if both packages are installed:
cd xpdf/var/log/packages
#note that you should NOT use a leading / before xpdf
ls
#you can see that both the xpdf and lesstif packages are installed here
cat * | less
#to see all of the files that came with the packages

Now, let's turn this into a module:

cd ~/temp/modwork/
dir2xzm xpdf xpdf-3.02pl6-i486+deps.xzm
#copy it to your /porteus/optional folder
cp xpdf-3.02pl6-i486+deps.xzm /mnt/sdXN/porteus/optional/

Now, you have a module with two slackware packages inside it, and whenever you activate it, everything you need for xpdf to run will be injected into the live filesystem.

Running 32bit modules on 64bit Porteus Edition

The 64-bit edition of Porteus has support for executing 32-bit applications built into the kernel.
However, applications which are not compiled statically still need 32-bit shared libraries to run.

If you want to be able to execute 32-bit applications on 64-bit Porteus, then you have to add 32-bit libraries to your system.

Please download and activate comapt32-libs module from this location:
http://dl.porteus.org/x86_64/current/extras/

[GUIDE] The Bleeding Edge

A short guide on compiling source code and making it into a module.
By Nadim Issa

Compiling source code is a task that most of us will run into while using a linux based system. This guide is intended to help those who are not too familiar with this. The source code that I will be using here is for a simple text editor called nano. This can be downloaded from http://ftp.gnu.org/gnu/nano/. The version should make no difference.I would venture a guess that to be on the bleeding edge you need to be running the latest version of all software you use. Sometimes the wait for someone else to build the module (or the package you need to install this software), will leave you on the coagulated side of things.

Usually, the first thing to do after downloading your source is to unpack it:

tar -xvvf nano-2.3.0.tar.gz

Don't laugh at my flags. I use them for every decompress I come across-- gz's,bz2,ect..

Some of you might say "What is all this mess I just unpacked on my computer?" It is the combination of documentation and source code. There are two key documents to us here, most applications I have compiled have these two files: INSTALL and README. These two files usually have all the information and prerequisites for compiling. Now the sweet and simple is we just issue the following commands.

cd nano-2.3.0
./configure
make
make install


This will work fine if you want to install it to your system root. But I wanted to share a couple of pointers and we are also going to make this into a module.

On the first try I will use the following code:

cd nano-2.3.0
./configure --prefix=/mnt/sda6/nano-os CFLAGS="-Os -fPIC -pipe -fomit-frame-pointer"
make -j4
make install


That looks like a lot of gibberish. Well don't worry it is pretty easy to understand.

The second line of that code calls a configuration script, and that configuration script creates the makefiles that the next two commands use to build the executable. By passing the --prefix= code we are changing the root of the install. We are doing this so we can isolate, easily, the executable,libs and documentation, of the application we are building.

After that we have the CFLAGS.  These options are used for telling the compiler how we are going to optimize our code, mess with it. Our first option in the CFLAGS is a -Os (that is a minus capital O not the number 0, followed by an s). This is used for identifing the optimization level. There is also Os, O1, O2, and O3. The Os typically opitimizes for code size, so it tries to produce the smallest executable. The O3 is usually used to optimize for speed of execution. O2 is the default that is used if nothing is specified. Another thing to remember is that just because the code is smaller doesn't mean it will execute faster.

The -fPIC is specific to the x86_64 architecture, and enables "Position Independent Code". This is well beyond the scope of this document. If you want more info on that, this document made a lot of sense to me: http://www.technovelty.org/code/c/amd64-pic.html

The -pipe forces inline instructions to do just that.

The -fomit-frame-pointer will make it very difficult to debug with traditional debuggers. This usually results in faster executing code.

The next line is make -j4.  Make starts the compile process and the -j4 tells the compiler to compile 4 threads simultaniously. This results in faster build times. How do you know how many threads to compile simultaniously? The usual formula is (number of cpu cores) X 2 + 1. For programs this small it doesn't make a difference but for larger projects it can usually result in hours of compile time difference. Occasionally you will run into errors while compiling. You can make it through some of the softer errors by issuing a "make -k -j4", the -k forces it through some errors.

Finally, the make install instruction just places all the compiled executables, library files, and documentation in the specified place.

Now I'll compile with a different set of flags, placing the resulting files in a different directory:

cd nano-2.3.0
./configure --prefix=/mnt/sda6/nano-o3 CFLAGS="-O3 -fPIC -pipe -fomit-frame-pointer"
make -j4
make install

Here are the results of those two compiles:

/mnt/sda6/nano-os/bin/nano 222672 bytes
/mnt/sda6/nano-o3/bin/nano 246974 bytes

Here it is clear that the -Os cflag resulted in a smaller executable.

Now all that is left to do is strip the executable to reduce its size. This step is optional. The first time I ever ran into this, I was compiling a kernel for a "Sun Ultra 80" with the Sparc cpu's.  For some reason the kernel had to be less than 4 mb, to load properly. So for some systems this step is essential. The way this reduces the size of the executable by removing unneeded symbols in the code. You can also do this to shared libs. This can lead to debugging issues.

strip --strip-unneeded /mnt/sda6/nano-os/bin/nano
strip --strip-unneeded /mnt/sda6/nano-o3/bin/nano

resulting sizes:

/mnt/sda6/nano-os/bin/nano 192240 bytes
/mnt/sda6/nano-o3/bin/nano 216784 bytes

I like to use the smaller executables. So I will use the nano-os directory for my module creation. I remove extra stuff that I don't feel is needed to include in my module. I remove the man files, info files, and the locales. Remember to keep what is important to you. Also with this particular package it needs to be installed to /usr/local/ so make sure you create these directories and move the appropriate files in place. So now the /mnt/sda6/nano-os/usr/local/ is where the prefix should have been. Now we build our module using the dir2xzm.

cd /mnt/sda6
dir2xzm nano-os nano-2.3.0.xzm


This is our final module size:

/mnt/sda6/nano-2.3.0.xzm 106496 bytes


At this point share with your friends, upload it to internet, do whatever you wish with it. You can also find more info on CFLAGS at http://en.wikipedia.org/wiki/CFLAGS
This will help keep you at the bleeding edge with software, on porteus.

To view the discussion about this article, or to make a suggestion or ask a question about it, visit this thread on our forum.

Compiling a module from source

Compiling a module from source code

You should be aware that there are many different forms of source code and not all source code follows the conventions stated here. This tutorial gives a general idea about how to compile a module from source code for a majority of sources (as at 2010) and is by no means a comprehensive study of compiling from source code. You should first know the basics of modules and understand how they are created.

When you download source code it will generally come in the form of a filename.tar.gz or a filename.tar.bz2 compressed file. For the following example the source package has the name ImageMagick-6.5.2-5.tar.gz. It was download to a folder in the home directory called Downloads. Here is a list of the basic commands needed in order to make a quick and dirty module for your own use only. If you plan on making an official module that will be uploaded to the Porteus repository please follow the rules and guidelines below.  These are the general rules provided by the creator of Slax, Tomas M, and will be largely adopted for use in Porteus.

Quick and easy compiling for personal use:

Generally after you unpack a source file there is a text file inside called INSTALL or README. These usually contain the information needed to build the package.

Set the package variable

# PKG=ImageMagick-6.5.2.5
# TMP=/tmp/$PKG

Make a temp working directory

# mkdir  $TMP

Change directory to source file

# cd ~/Downloads

Unpack the tarball:

# tar zxvf ImageMagick-6.5.2-5.tar.gz
# tar xvjf  ImageMagick-6.5.2-5.tar.bz2

Enter into the source directory

# cd ImageMag*

Start the configure process

# ./configure --prefix=/usr

If you get an error message saying there is no configure file then you may need to install a package called autoconf and run it. The next step is to check that the Makefile contains the  variable DESTDIR. This is required in order to make a module. If it does not exist then you may get away with using the --prefix= switch in the configure process.

# grep "DESTDIR" Makefile

If you get no output from this then reconfigure using:

# ./configure --prefix=$TMP

Now start the make process

# make

Now create the package. The || means if the first command fails jump to the second.

# make install-strip DESTDIR=$TMP || make install DESTDIR=$TMP

Strip the binaries

# pushd $TMP
# find . | xargs file | grep "executable" | grep ELF | cut -f 1 -d : | xargs strip --strip-unneeded 2> /dev/null
# find . | xargs file | grep "shared object" | grep ELF | cut -f 1 -d : | xargs strip --strip-unneeded 2> /dev/null

If the package requires a menu entry you should now find a suitable icon image and place it in /tmp/ImageMagick-6.5.2.5/usr/share/applications

# mkdir -p $TMP/usr/share/pixmaps
# cp /path/to/imagemagik.ico $TMP/usr/share/pixmaps

Create the menu entry and desktop file now

# mkdir $TMP/usr/share/applications
# cat >> $TMP/usr/share/applications/imageMagick.desktop << EOF
[Desktop Entry]
Type=Application
Encoding=UTF-8
Name=ImageMagick
Icon=imagemagick.ico
Terminal=false
Categories=Graphics
EOF

Pop back out of the source directory and create module

# popd
# dir2xzm $TMP $PKG.xzm

If you compiled with the DESTDIR=$TMP switch then you will need to check that inside your $TMP directory you the normal filesystem folders such as /usr /bin. If not then adjust your dir2xzm command so that the base folder contains the correct folders. It may be something like: dir2xzm $TMP/$TMP /tmp/mymodule.xzm

That should be all you need to create your module. Remove your tmp directory and you are done. Please remember that this module should only be used for your own personal use. It may not run smoothly on someone else's system.

Module creation rules

Every Porteus module contains all files and directories with full path needed for the package to function. For example, a module with bash (the binary and some man pages) would look like this:

/bin/
/bin/bash
/usr/
/usr/man/
/usr/man/man1/
/usr/man/man1/bash.1

Since Porteus is designed to be a lightweight portable system the size of your module should be as small as possible. Uncompress all archives which may be safely left uncompressed (for example man pages, because LZMA will compress them far better), delete all files which are not needed to run the software (for example unneeded documentation, unused sounds, png/jpg images, unneeded translations from /usr/share/locale) and strip all unneeded symbols from binaries.

If you compiled the module from source codes and you used a script then provide the build script, which is used to create the module. The build script must handle the whole module creation. Any manual work (copying / deleting files, etc) aside the build script is not allowed. The script serves as a documentation as to how to create your module; moreover it makes it easy to take over your module in the case you stop updating it. Copy the build script to your module to:

/usr/src/porteusbuilds/your_module_name.porteusbuild

When you compile the software, make sure to use correct cflags and parameters. The following is recommended in order to use i486 instructions (which provides the best backward compatibility), but tune the performance of the code as if the target architecture was i686.

CFLAGS="-O3 -march=i486 -mtune=i686" ./configure --prefix=/usr --build=i486-Slackware-linux

Never include any existing files from Porteus in your module, even if you modified them. In other words, your module should never 'overwrite' any existing file in Porteus, unless you have a sensible reason to do so. It can make your module incompatible with newer Porteus versions and it can cause problems with modules from other users. If you really have to overwrite a file in Porteus, (for example in order to append a new path to /etc/ld.so.conf), write a startup script instead, which will modify (update) the particular file, instead of overwriting it by your module.

Example startup script to delete one line from ld.so.conf and add a new one at the end:

#!/bin/bash
sed -i -r '\;/usr/local/lib;d' /etc/ld.so.conf
echo '/opt/kde/lib' >> /etc/ld.so.conf

Here is a sample list of few files which should never be included in your module:

/etc/init.d
/etc/rc.d/rc.S
/etc/rc.d/rc.M
/etc/rc.d/rc.K
/etc/rc.d/rc.local
/lib/modules/2.6.x/modules.dep
/lib/modules/2.6.x/modules.alias
/etc/ld.so.conf
/etc/ld.so.cache
/etc/passwd
/etc/group
/etc/shadow