HowTo: Debugging with GDB

Since some month I am working with MMORPG emulators again, written in C++. Because of this I needed a powerful debugging tool and so I chose GDB.
I will show you some nice-to-knows for debugging with GDB on a simple code sample.

I recommend at least basic knowledge of C++, Linux and debugging to understand most of the things explained in the following part.

Assume we have this small C++ code block:

int MyDivide(int, int);

int main()
{
    int x, y, result;
    x = 10, y = 2;
    result = MyDivide(x, y);
    x = 4; y = 0;
    result = MyDivide(x, y); // Division by zero
    return result;       
}                        

int MyDivide(int _x, int _y)
{   
    return _x / _y;
}

Now put this in a file, called “divide.cpp”, and compile it to a binary, called “divide”:

$ g++ -g divide.cpp -o divide

After doing this, let us start with the GDB magic and load the “divide” binary:

$ gdb divide 
...
(gdb)

Everything is prepared, so here are some useful things you can do with GDB:

  1. Start application:
  2. (gdb) run
    Starting program: /home/dennis/Entwicklung/Misc/divide 
    
    Program received signal SIGFPE, Arithmetic exception.
    0x0000000000400513 in MyDivide (_x=4, _y=0) at divide.cpp:15
    15          return _x / _y;

    In our case the startup of the application results in a crash, due to division by zero.

  3. Backtraces:
  4. Because of the crash, we now can get more detailed information with the backtrace.

    (gdb) bt
    #0  0x0000000000400513 in MyDivide (_x=4, _y=0) at divide.cpp:15
    #1  0x00000000004004f9 in main () at divide.cpp:9
    (gdb) bt full
    #0  0x0000000000400513 in MyDivide (_x=4, _y=0) at divide.cpp:15
    No locals.
    #1  0x00000000004004f9 in main () at divide.cpp:9
            x = 4
            y = 0
            result = 5
  5. Breakpoints:
  6. (gdb) break divide.cpp:7
    Breakpoint 1 at 0x4004ca: file divide.cpp, line 7.
    (gdb) break divide.cpp:9
    Breakpoint 2 at 0x4004ea: file divide.cpp, line 9.
  7. Step debugging:
  8. (gdb) run
    Starting program: /home/dennis/Entwicklung/Misc/divide 
    
    Breakpoint 1, main () at divide.cpp:7
    7           result = MyDivide(x, y);
    (gdb) next
    8           x = 4; y = 0;
    (gdb) continue
    Continuing.
    
    Breakpoint 2, main () at divide.cpp:9
    9           result = MyDivide(x, y); // Division by zero

    With the “next” command, GDB will go to the next line, which is executed by the application. As opposed to this the “continue” command will simply continue the application.

  9. Variable manipulation:
  10. (gdb) print y
    $1 = 0
    (gdb) set y = 1
    (gdb) print y
    $2 = 1
  11. Exiting application:
  12. (gdb) quit

I hope this tutorial helps you to work with GDB and to get some personal achievements with debugging your applications.

If you have feedback, regards, corrections or questions please let me know and do not hesitate to comment!

HowTo: Convert Subversion to Git

To make a start I can say that I personally prefer using Git as Version Control System. I use Subversion and Mercurial for some reasons too, but I do not really like to work with them.

Because of this I wanted to convert some Subversion repositories to Git. If you also have planned to do this, the following may help you.

Three different ways for the conversion:

  1. Copying the source code:
  2. The easiest and fastest way is simply copying the source code into an empty Git repository. For me this solution was impracticable, because you will loose the history.

    # Create and switch to new folder
    mkdir $GIT_REPO
    cd $GIT_REPO

    # Copy all files from the Subversion directory
    # Use -C option to ignore .svn directories
    rsync -C $PATH_TO_SVN_REPO/* $PATH_TO_GIT_REPO/

    # Do the Git steps
    git init
    git add *
    git commit -asm "Initial commit"

  3. Using git-svn:
  4. The next way to convert the repository is using git-svn. With git-svn you are able to adopt all branches and the whole history. The only unsuitable thing is converting tags, because they are created as branches by default and you need to switch them manually afterwards.

    # Install the software
    (sudo) apt-get install git-svn

    # Make and switch to new folder
    mkdir $GIT_FOLDER
    cd $GIT_FOLDER

    # Convert the repository
    git svn clone $SVN_URL --no-metadata --stdlayout .

    # Update the repository
    git svn fetch

  5. Using svn2git:
  6. Svn2git is a utility which uses git-svn internally to do the real conversion, but in addition it does some more jobs around this to perfect the system. With this you will have a accurate converted repository. It uses Ruby, so you need to install some additional packages.
    You can use svn2git for some special actions and custom purposes. If you need some more information you can take a look at the README.

    # Install the software
    (sudo) apt-get install git-svn ruby rubygems
    (sudo) gem install svn2git

    # Make and switch to new folder
    mkdir $GIT_FOLDER
    cd $GIT_FOLDER

    # Convert the repository
    svn2git $SVN_URL

    # Update the repository
    svn2git --rebase

I hope this tutorial helps you to save some time and troubles. If you have feedback, regards, corrections or questions please let me know and do not hesitate to comment!

I will probably write something about Version Control Systems, like a comparison or Cheat-Sheets, in other blog posts.

MySQL Security Tips

In this post I will show up some dangers of the MySQL-Server usage you should care of. These are the first steps to enhance the security of your database server, because with this you are able to limit the access and the privileges.

I recommend at least basic knowledge of MySQL to understand most of the things explained in the following part.

Some dangers that people underestimate:

  1. Empty passwords:
  2. The first thing you should check and avoid on are empty passwords. On worst circumstance everyone can log on to your MySQL-Server!

    With a simple SELECT-Statement you can check this and identify all users without any password.

    SELECT User FROM mysql.user WHERE Password = "";

    To solve this security issue you need to assign a password to the user with following UPDATE-statement.

    # Replace $PASS with the password
    # Replace $USER with the user

    UPDATE User SET Password=PASSWORD("$PASS") WHERE User="$USER";

  3. Careless hosts settings:
  4. CREATE USER 'user'@'%' IDENTIFIED BY 'password';

    This is a bad example for creating an user in MySQL. The percent sign means that you can connect with this user from any host, what can have serious security impact.

    There are better ways to do this. You can set exact IP-addresses, host-names or use wildcards for both ones to limit the connection possibilities of offenders significantly.

    CREATE USER 'user'@'localhost' IDENTIFIED BY 'password';
    CREATE USER 'user'@'10.11.%.%' IDENTIFIED BY 'password';

  5. Lack of permissions:
  6. The MySQL permission system is a very time consuming section. If you want to manage this correctly, you need to know all necessary permissions to every table in every database for every single user!

    Because of this people often want to make their life easier and so they use a simple GRANT-statement, which sets all permissions.

    GRANT ALL PRIVILEGES ON *.* TO 'user'@'host';

    But, as you can imagine, this can have a very bad influence on your security. If your software is for example vulnerable for SQL-Injection and you do not know this, attackers can simply drop all databases!
    To prevent this I recommend setting only the required privileges to the users.
    If you need an overview about all privileges you should have a look at the MySQL privileges section.

    GRANT SELECT,INSERT,UPDATE,DELETE ON *.* TO 'user'@'host';

    A better way than only setting the limited permissions to all databases and tables is, setting different privileges for every table!

    GRANT SELECT,INSERT,UPDATE,DELETE ON DB.TABLE TO 'user'@'host';

  7. Bind-address in my.cnf file:
  8. The bind-address restricts all connections to the MySQL-Server to a given IP-address.

    In most cases you will see the line below commented out in the my.cnf file. This is necessary if you need to connect to the MySQL-Server from different hosts.

    #bind-address = 127.0.0.1

    But if you only need access to the MySQL-Server on localhost it is better and more secure to use this as bind-address, because the MySQL-Server does not listen on external connections then.

    bind-address = 127.0.0.1

I hope this tutorial helps you to secure your MySQL-Server. In fact there are much things you can do to make it more secure, but like I said this is a first step to enhance it.
If you have feedback, regards, corrections or questions please let me know and do not hesitate to comment!

Gitlog Parser

Sometimes I needed a possibility to convert the output of the `git log` command in a useful format. Because of that I have written a tool in Perl, which is currently able to convert this output to different file formats.

Here is a short overview of this tool:

  1. Link:
  2. https://github.com/DoesntMatter/GitlogParser

  3. Get it:
  4. # Clone from github
    $ git clone git://github.com/DoesntMatter/GitlogParser.git

  5. Current supported formats:
    • SQL-Dump
    • RSS-Feed
    • HTML-Page

  6. Documentation:
  7. You can find the documentation in the README file of my github repository.

  8. Sample usage:
  9. # Change to GitParser directory
    $ cd YOURPATH/GitlogParser
    # Show help
    $ ./scripts/gitlog_to_rss.pl --help
    # Execute script for RSS output
    $ ./scripts/gitlog_to_rss.pl --repo PATH-TO-GIT-REPO


    The –repo option is the mandatory option to state the git repository. Of course there are other options to customize the output for your purposes. For example you can specify the title of your RSS-Feed with the –title option. If you want to know more about the other options please take a look at the documentation or use the output of the –help option.

  10. Sample output:
  11. RSS
    HTML
    SQL

This tool is not finished completely and I will probably implement support for more file formats. If you have suggestions or any feature request please do not hesitate to contact me via github!

HowTo: MySQL Performance Tuning

At first I want to clarify that this is only a guideline for tuning your MySQL Performance, due to the fact that every MySQL Server is used for custom purposes and needs. There can be huge differences for application use, like READ-heavy or WRITE-heavy databases.

I recommend advanced knowledge of MySQL to understand most of the things explained in the following part.

This post includes some basic tips to increase your MySQL Server performance. Maybe I will write some other posts about more specific topics, like buffers or caching.

Tips for MySQL performance optimization:

  1. Storage Engine:
  2. In my last post I described and explained the differences, advantages and disadvantages of the most popular MySQL storage engines: MyISAM vs. InnoDB. The choice of the storage engine that fits best to your needs is very important. If you have done the correct selection this can have huge performance enhancement and more features too!

  3. Table optimization:
  4. Another common problem is table fragmentation, which can have great performance impact too. The main problem with fragmentation is bad ordering of indexes on disks, which results in slower query processing due to I/O issues.
    On my github profile you will find a tool for checking and optimizing your tables: MySQL fragmentation tool.

    Here is a short explanation how you can use this:
    # Show table count of fragmented tables in your databases
    ./mysqlfragfinder.sh --user $USER --password $PASS --check
    # Show fragmented tables in your databases
    ./mysqlfragfinder.sh --user $USER --password $PASS --check --detail
    # Optimize all tables
    ./mysqlfragfinder.sh --user $USER --password $PASS
    # Optimize all tables of one specific database
    ./mysqlfragfinder.sh --user $USER --password $PASS --database $DB

    So for better performance you should optimize your tables regularly. But beware of using this with productive databases, because optimizing a table will lock it!

  5. Tools for analysis:
  6. This section is only for advanced users, because you need experience in MySQL server configuration. There are some tools with which you can analyze your MySQL server very well. With the help of them, you are able to find possible configuration failures or optimization targets. You find them on my github account too: MySQL Tuner and MySQL Report.
    MySQL Tuner itself displays tips and data in its output, whereas MySQL Report displays more detailed data. Besides MySQL Report has a really great guide in which their output is explained: Guide for MySQL Report.
    Both tools are really helpful, but they will not do the configuration for you. So be sure you know what you are doing if you follow their tips!

    My recommendation is taking a closer look on following topics:

    • Slow queries
    • You should ensure that the slow query log is enabled in your configuration. If you have slow queries try to get rid of them immediately! There is nothing worse than queries, which keep your MySQL server stuck.

    • Key buffer
    • The key buffer is very important for your indexes and you should always guarantee that there is enough free buffer space left. With the help of this the indexes of the tables are processed in memory and not on disks. So if you use them this will result in a huge performance boost.

    • Query cache
    • The query cache is used by SELECT-statements which for example get executed very often and their result changes very rarely. If they are cached the result comes out of the memory, which is much faster than from disk. So you should set the query cache value high enough to profit from this advantage.

    • InnoDB buffer pool
    • This is only available within the InnoDB storage engine and it should be set to the size of all InnoDB tables plus a margin, so that they can expand. The buffer pool is one huge advantage of InnoDB, because it reads additional data from memory too.

    Of course most of the options and variables depend on your custom purposes and vary due to that. Because of that you need to check the special output of the tools for possible problems by yourself.

I hope this tutorial gave you some hints and tips to optimize your MySQL server. If you have feedback, regards, corrections or questions please let me know and do not hesitate to comment!

If my time permits and there is some demand, I will write some posts on more specific topics of this post too.

MyISAM vs. InnoDB

Before talking about MySQL performance tuning, it has to be clear which MySQL storage engine should be taken. I will talk about the two most popular storage engines: MyISAM and InnoDB.

I recommend at least basic knowledge of MySQL to understand most of the things explained in the following part.

At first it is important to mention that there is no storage engine which is best of all. Every storage engine has its advantages and disadvantages, but if you clarify your needs correctly you will find the one that fits the best.
In fact you do not have to use one storage engine for your whole database, but you are able to combine the advantages by using both of them for different tables. So probably the way to the best performance and usability is using MyISAM and InnoDB!

Description of the most important MySQL storage engines:

  • MyISAM
    1. Features
    2. This storage engine is used with simple and easy database models and purposes in general. For me the most important feature of MyISAM is full-text search.

    3. Advantages
    4. It is the default storage engine of MySQL, because its simplicity is a great advantage. You do not have to take care of any complex database models and can easily create simple table designs.
      Another important point is the full-text indexing, already mentioned as feature. You are able to use this for search engines in forums, websites or web-shops.

    5. Disadvantages
    6. Due to the simplicity there are many things which MyISAM lacks, even if it is an advantage. MyISAM is missing of data integrity features such as foreign keys.
      Also the missing transactions feature is bad, because queries can mess up your complete table content!
      Another negative point is that this storage engine does table locking. This will surely reduce performance with WRITE-heavy tables.

    7. Fields of use
    8. MyISAM is the best storage engine for beginners or simple purposes, but I do not recommend using it with WRITE-heavy tables, due to table locking and the lack of data integrity and transactions.
      All in all MyISAM is mostly considered to be used in relation with tables of websites, content management systems, blogs, small search engines and so on. As you can see, these examples are all READ-heavy and so it is mostly used in this field.
      Of course this a general statement which depends on the size of the software or on specific tables. There are some cases on which other storage engines have better performance results with SELECT-statements, but this is not the standard.

  • InnoDB
    1. Features
    2. This storage engine is used with more complex database models and purposes in general. There are three main features which InnoDB has: Relational database design with foreign keys, transactions and row-level locking.

    3. Advantages
    4. In my opinion the most important advantage of InnoDB is the transaction feature. With this you can ensure the correctness of your data, even if a data manipulation query gets interrupted.
      Besides it supports row-level locking within tables, which is very important for the performance of WRITE-heavy tables.
      Another advantage is the data integrity with the help of foreign keys. With this you can build relational database models, needed for complex use.
      Moreover InnoDB not only keeps indexes, but also frequently accessed data in memory, what is possible with the buffer pool. Of course this reduces I/O on disks and due to this increases perfomance a lot!

    5. Disadvantages
    6. In opposite to MyISAM this storage engine does not support full-text indexing, which prevents you from doing full-text searches.
      Due to the fact that InnoDB maintains data integrity, designing database concepts will be more time-consuming.
      Another thing is that it also needs much more system resources, especially RAM. Only with this it is possible to guarantee good performance.

    7. Fields of use
    8. To summarize InnoDB is a storage engine for advanced users with databases which contain sensitive data.
      It is recommend to use it with WRITE-heavy databases, because of the transaction feature and data integrity.
      Nowadays you can use InnoDB for READ-heavy databases too, because the times of slow performance are really gone. But this gets only important with huge applications or websites with huge count of visitors and changes.

I hope the differences and advantages of both MyISAM and InnoDB get clear after reading this article. If you have feedback, regards, corrections or questions please let me know and do not hesitate to comment!

Besides this was the first step to and the base of my MySQL performance tuning post, which will come soon.

HowTo: Install Percona Server

As I promised in the last post, I am going to show you how to install the MySQL-Server solution of Percona.

There are two ways of installing Percona Server: Via package manager or from source. Maybe some of you think that it is not needed to install Percona Server from source and they are right. It is not really needed, but it has some advantages which you will see later on. For the sake of completeness I will show both ways.

This HowTo is based on my personal purposes and needs, so there are other possibilities to get Percona running for sure.
My favourite operation systems are Debian and Ubuntu and I currently use Percona-Server v5.5.15-21.0, but will probably upgrade in some weeks.

I recommend at least basic knowledge of MySQL and UNIX to understand most of the things explained in the following part.

The two ways of installing Pecona Server:

  1. Installion via packet manager
  2. The only thing you have to do is getting the signed key of Percona and adding the repositories to your source list. With this you are able to install the software automatically via package manager.
    You will need root access to do following actions.

    # Get the key
    $ gpg --keyserver hkp://keys.gnupg.net --recv-keys 1C4CBDCDCD2EFD2A
    $ gpg -a --export CD2EFD2A | sudo apt-key add -

    # Add the repositories
    # Please replace $RELEASE with your current OS release below
    $ echo -e "\n# Percona Server\ndeb http://repo.percona.com/apt $RELEASE main\ndeb-src http://repo.percona.com/apt $RELEASE main" >> /etc/apt/sources.list

    After that you are ready to install Percona Server yet!

    $ apt-get install percona-server-server-5.5

  3. Installation from source
  4. This version is more complex, but you have the possibility to install the application for custom purposes.
    You will need root access to do following actions.

    # Add mysql user
    $ useradd -s /bin/false -b /opt/mysql -d /opt/mysql -m mysql

    # Install MySQL client
    $ apt-get install mysql-client-5.1

    I prefer installing the client previously, because it already creates the MySQL configuration files like the my.cnf in /etc/mysql/ which is not created by this installation method. Indeed Percona provides different versions of my.cnf files for different purposes, but I will topic this in another tutorial.

    Now we can start with the basic installation.

    # Install required packages
    $ apt-get install automake libtool g++ ncurses-dev bison

    # Get and extract Percona source
    $ wget http://www.percona.com/redir/downloads/Percona-Server-5.5/Percona-Server-5.5.15-21.0/source/Percona-Server-5.5.15-rel21.0.tar.gz
    $ tar xvfz Percona-Server-5.5.15-rel21.0.tar.gz
    $ cd Percona-Server-5.5.15-rel21.0

    # Prepare build
    $ sh BUILD/autorun.sh
    $ ./configure --without-plugin-innobase --with-plugin-innodb_plugin --prefix=/opt/mysql

    # Create directory for logging
    $ mkdir /var/log/mysql
    $ chown mysql:mysql /var/log/mysql

    I prefer to install MySQL in /opt/mysql, so you can specify this with the –prefix option. The two other options trigger the usage of XtraDB instead of the build-in InnoDB storage engine. I wrote something about this in my last article of Percona Server. For increasing InnoDB performance it is very important to use this!

    # Build the sources
    $ make -j
    $ make install

    # Install basic database
    $ cd /opt/mysql
    $ ./scripts/mysql_install_db --user=mysql --basedir=/opt/mysql --datadir=/opt/mysql/data --verbose --log-error=/tmp/mysql.error.log --defaults-file=/etc/mysql/my.cnf

    # Set MySQL user privileges
    $ chown -R mysql:mysql /opt/mysql

    If you followed these steps correctly you should have a database which is almost ready to run. We will now create some files to be able to start and stop MySQL server easily and provide correct startup and shutdown behaviour with OS actions.

    # Use Percona support files
    $ echo "/opt/mysql/support-files/mysql.server start" > /etc/init.d/mysql_start
    $ echo "/opt/mysql/support-files/mysql.server stop" > /etc/init.d/mysql_stop
    $ chmod 755 /etc/init.d/mysql_*

    # Link them to rc directories
    $ ln -s ../init.d/mysql_start /etc/rc2.d/S19mysql
    $ ln -s ../init.d/mysql_stop /etc/rc0.d/K21mysql
    $ ln -s ../init.d/mysql_stop /etc/rc6.d/K21mysql

    The last thing you have to do before startup are small adjustments to your MySQL configuration. There have to be some necessary changes to get this working correctly. The advantage of my.cnf is that it has an include path for custom configuration files. So let us use this due to the fact that the content of these files overwrite my.cnf options.

    # Add custom paths
    $ echo -e "[mysqld]\n\nbasedir = /opt/mysql\ndatadir = /opt/mysql/data\n" > /etc/mysql/conf.d/mysql_additon.cnf

    # Required XtraDB config
    $ echo -e "innodb_file_per_table = 1\ninnodb_file_format = barracuda" >> /etc/mysql/conf.d/mysql_additon.cnf

    I just set the correct base- and data-dir of MySQL, because I prefer this setting. It is up to you how handle this. The XtraDB configuration settings I did are really important! The first one (innodb_file_per_table) is set to 1 and used for reducing file access and therefore reducing I/O. This becomes more important if you have slow disks, but is recommended by me anyways. The second one (innodb_file_format) is set to barracuda, which is the only supported table format for XtraDB.

    Now you should be able to start and stop your MySQL server with following commands:

    # Start
    $ /etc/init.d/mysql_start
    # Stop
    $ /etc/init.d/mysql_stop

I hope this tutorial helped you to install and configure your Percona Server. If you have feedback, regards, corrections or questions please let me know and do not hesitate to comment!

Keep in mind that there will follow guides for MySQL performance tuning and a backup solution for InnoDB and MyISAM tables.

Percona Server

A very interesting topic of server administration is the choice of a nice MySQL-Server for high performance purposes and solutions.
Many people underestimate this choice, which is surely a huge mistake! The correct research before setting up your database server is the first step into a really good direction.

Let us talk about the MySQL-Server solution of Percona a bit. This is my personal choice and I will give you some basic background information and nice to knows about it.

I recommend at least basic knowledge of MySQL to understand most of the things explained in the following part.

Most important reasons to choose Percona instead of “standard” MySQL solution:

Before I will go into detail, I want to show you a graphic from Percona.com which shows where the story goes.

  1. Stability

  2. Stability here does not mean that the server is free of crashes, because you can truly say that this counts for all MySQL versions too. It means the stability on which Percona answeres database questions or executes queries and prepared statements. This really is a must have if you want to run powerul applications without lags or huge overload. Of course the benefit depends on the usage of your database server (READ- or WRITE-heavy), but this can never be a disadvantage. In the graphic above you can clearly see the amazing difference between MySQL- and Percona-Server!
    In my experience, with WRITE-heavy databases, it is really needed to guarantee a stable and consistent service.

  3. Performance
    • Over-all
    • As you can see in the graphic above and read in the article on Percona site, it is 40% faster than MySQL! To repeat this: Without doing anything you simply get much more efficiency! Now imagine this combined with Server SDDs and more performance changes you did manully. I think every additional word is wasted here.

    • InnoDB/XtraDB
    • As if the over-all performance was not enough, all InnoDB users will be happy to hear that Percona has been doing a great job in tuning InnoDB perfomance too. They developed a high performance InnoDB clone, called XtraDB. You can see the huge performance impact in following graphic from Percona.com:

There are more reasons to use Percona Server of course, but I think the most doubts are killed by explaining stability and performance advantages above. I hope you enjoyed reading this and maybe switch to Percona Server too. If you have feedback, regards or corrections please let me know!

Any time soon a tuturial for installing and setting up Percona will follow. Additionally I will show you how to optimize and tune your Percona- or MySQL-Server on your purposes in another tutorial.

Getting started

Hey guys!

This blog is work in progress, so you won’t find much information here at the moment.
I will write some articles and create some pages in the next weeks.

The topics you will find here will mostly contain guides, tips, scripts and more about useful software I worked with.

I look forward to creating some interesting stuff here and getting some feedback.

Best Regards
DoesntMatter