Grief

We encounter grief at various stages in life. It could be personal loss,disappointment or loss of a beloved one. When at each stage we encounter grief it changes a part of us. However small cause may be it changes I guess at least one neuron connection. I have had numerous such encounters. The most significant is the one when I lost my grandfather. Today I lost a second one and just watching him lie there still evoked the trauma that I underwent when I lost my first grandfather. At some moments I could see him lie there and not my second grandfather. It was in way creepy. But in few moments I realized it was the sign of how much I missed him and the size of his influence on me. I realize this will also change me a bit. Teaching me something that I would realize at a later stage. Grief changes and teaches.

xtrabackup incremental backups

Tags

, , , , , ,

We wanted to optimize and test backups for one of our new large scale setup before we could finalize on the backup plan. Our challenge was the data volume was almost 30x more than our normal volumes. Since this involved large volume of data we thought this might be a good candidate to test incremental backups. We wrote a wrapper script to save the states between full backups and incremental backups and did some tests with smaller data sets. It worked perfectly fine. The version of xtrabackup we were testing was 1.6.2.

The incremental backups were completing well within 30minutes. We set out to test what would happen if the amount of incremental diff was large. While doing this test, we started getting backup failures with the following error

20110726_225312.log-110727 00:56:10 innobackupex: Starting to backup .frm, .MRG, .MYD, .MYI,
20110726_225312.log-innobackupex: .TRG, .TRN, .ARM, .ARZ, .CSM, .CSV and .opt files in
20110726_225312.log-innobackupex: subdirectories of ‘/var/lib/mysql’
20110726_225312.log:innobackupex: Error: Broken pipe at /var/backups/xtrabackup/bin/innobackupex line 336.
20110726_225312.log-Backup failed.
20110726_225312.log-Deleting bad data directory…
20110726_225312.log-Done. Removed /ddmysql/backups/20110726_225312

Initially we thought this was a one of case and made quite a few reruns. It was failing consistently. We wanted to get to the bottom of this. This was important for us to finalize the backup plan. We ran the innobackupex script through a debugger to find out where it was failing. The broken pipe error was getting triggered when the script was trying to send a ping to the mysql server so that the connection doesn’t timeout after the number of seconds mentioned in “mysql_keep_alive_timeout”  variable. This was set to around 1800. Our conclusion was when the incremental was of smaller size, the script never got to send the timeout ping. We reran the script with a much larger timeout value for a larger incremental data set and the backup was completed successfully.

The script seems to lose the connection handle during  course of execution and when it tries to ping on the handle it encounters a broken pipe. Wondering if there are other tried and tested ways to overcome this issue. Bug has been logged with the xtrabackup team to see if there are other options.

Dreams

Tags

, , , , , , , , ,

This is a phenomenon that is also applicable to our government schools in Tamilnadu. In our village  schools it is considered great if 5% of the students move up into high school from middle school. The world of these kids are so small that they are intimidated by thought of dreaming big. If they cross that chasm and dream big, they are faced with numerous hurdles that they eventually drop off somewhere along our system.  The Gingham team has got together to do our bit to expand the horizon of the kids to dream big and to urge the kids who have a dream to follow through. Please visit our website that we have setup for project "Gingham"  and let us know your thoughts. Even your smallest act of support by sharing/contributing/giving feedback will help a deserving kid to make his/her dream come true. Please help.

P.S -  This video was not created by our team but captures the essence so well.

Life Matters

Tags

, , , , , , , , ,

The number of “Aha!” moments that I have experienced in life are not that many. And I tend to forget those moments in course of time. I would like to narrate one such moment which I don’t want to forget and will not forget. Of late I have been conscious of the fact that life is passing by at break neck speed. Weeks and months whizz by in a cycle. home

As we enter each week we wait expectantly for the weekend and while we laze away the weekend, we start waiting expectantly for the next weekend. During these cycles we spend enormous amounts of time, energy and money to better our lives, our families and friends. We seem to add so much value to our organizations, our teams, our families. We rate ourselves so high that we even think that we are indispensible to the company, team or project during our moments of self pride and honor. We always keep thinking about the next big thing that is going to happen in our life. We wait expectantly gathering our resources while we pass away our current successes as if they are insignificant. We never reach a state where something is enough. There is always something better to pursue for in life. I totally agree that there is nothing wrong in having better expectations and goals. Today’s success is defined by status, money, power, position, fame and acclaim. That’s the mantra of modern day society. Anything that doesn’t matter to the modern day society is laid to rest by the wayside.

The moment made me reaffirm that there is more to life than the success criteria’s that we define for ourselves. Recently me and my friends visited few governments schools and homes in Cuddalore as part of a study. We were accompanied by one of my friend’s father to these places. In Cuddalore there is a home that is being run by the government just for girls who are orphans and whose parents are incapable of taking care of them. My friend’s father is a retried Head Master of one of the government schools. We reached the Girls home in a cab and he swiftly got out of his car being all excited to meet the kids. And what unfolded after that was totally out of the world experience for me. I could feel so much happiness and satisfaction by just watching the scene. The Girls were getting ready for their evening study and the moment they say him all hell broke loose. They all ran towards him, surrounded him , engulfed him showering with greetings and enquires about how he is doing and the smaller ones asking why he left. Words cannot describe the scene. The joy and the jumping is inexplicable.  The greetings and happiness was so blissful and warmth that I didn’t want the evening to end. Some even went to the extent of requesting him to come back. There were so happy to see their beloved teacher back in their hut even if it was momentary. This might sound like a cliché but we shall never understand the depth of it unless we experience it first hand. While coming back, I kept replaying the whole scene again and again in my mind. He has been their source of guidance and hope. He has touched the lives of not just his family / friends. He has inspired many children to greater levels. He doesn’t limit his circle of impact to only to people who can reciprocate. He has extended his knowledge to anyone and everyone He has crossed paths with. In His world the criteria of success is “Life Matters” , but not his. It is the Life of each and every kid that He comes in touch with is what matters. Even after His retirement, he doesn’t want to stop. Now the only question I keep asking myself is “What is my sphere of influence”.

Your compass is YOU

Tags

, , , , ,

We always have difficulty in making choices. There is always pressure on us deciding between 2 things (read anything)  even when the differences between the choices aren’t  that much. The problem of choice and its associated decisions can make us all stressed out and worrisome. The same can be said for feelings that are nagging at the back of our mind about something done to us or something that your friend forgot to do and ignored or something that was not done to your liking. I have always struggled with both, choices and nagging feelings. Our first instinct is always confrontation . In the case of choices we always confront ourselves with the choices and try to evaluate and decide which way to go. In the case of nagging feelings we might try to confront or blurt with the other person / persons involved about what we think or feel. It is always dangerous to do something when we are not sure what we want or what we actually feel. Forever I have struggled with these dilemmas. One of my friend whom I met recently inspired me to an idea that has worked so well for me in the past few months. It is very simple but very handy.

 

Observe your feelings about the choices or events over a period of time

 

Let me explain. We can break this down into simple 4 steps :

  • We have to have an item to evaluate or think over. It can be a choice , event or an experience.
  • Define what a favorable outcome will look like wrt to that choice/event/experience.
  • Give it some time, list the feelings and thoughts that you have about your point of contention over a period of time , say how do you feel about it in 2 weeks, how you feel in a months time based on available time.
  • Eventually by the end of your slotted time your feelings would have started to kick in enabling you to make a move in the right direction.

Decisions or choices made impulsively might not work in our best interests in the long run. The above approach has helped me decide upon during my numerous outrageous moments of dilemma. Let me know what works for you in your moments of choice.

Peace out !

running mysql with “memlock”

The documentation says that we need to run mysql as “root” user for it to be effective. The code also indicates the same thing where it tries to do a getuid call to check if the server is running as root user (sql/mysqld.cc:  if (locked_in_memory && !getuid())). The general rule of thumb is that we don’t want to run mysql or rather any database as root user since that leads to vulnerabilities.

Historically memlock seems to cause issues on some OS flavors where “mlockall” implementation is not stable. This is documented in mysql code with the below warning message.

The "–memlock" argument, which was enabled, uses system calls that are

unreliable and unstable on some operating systems and operating-system

versions (notably, some versions of Linux).  This crash could be due to use

of those buggy OS calls.  You should consider whether you really need the

"–memlock" parameter and/or consult the OS distributer about "mlockall"

bugs

On machines where only the database is hosted, if the memory sizing and parameters are set  optimally we might never want to use memlock.  It might be effective  to use “memlock” in machines where the database co-exists with  the application code and possibility of the database memory buffers getting swapped out is very high. Even on optimally sized standalone database machines due to various reasons (Jeremy has a nice blog post on this), there are chances of the database memory buffers getting swapped.

“memlock” seems to be a good option to pin the database in memory. But given the caveat that the servers needs to run as root user makes this option not usable in production since we don’t want to open up to vulnerabilities. Is there a workaround to use “memlock” in production or rather what would be the recommended way of using it.  

mount: /dev/sdb1 already mounted or /mysql busy

Tags

, ,

We added a 500GB 7.2K SATA/300 Hitachi Deskstar E7K500 16MB disk to one of our dev servers and partitioned using fdisk and formatted the partition with etx3. When we tried mounting the same, we got the following error :

[root@xyz user]# mount -t ext3  /dev/sdb1 /mysql
mount: /dev/sdb1 already mounted or /mysql busy

lsof didn’t provide any open files that might be linked to this problem or there was any “famd” running. Finally doing the following steps to remove the logical devices from the device-mapper driver helped us fix the problem.

[root@xyz user]# dmsetup ls
ddf1_44656c6c202020201028001510281f033832b7a2f6678dab   (253, 0)
ddf1_44656c6c202020201028001510281f033832b7a2f6678dab1  (253, 1)

[root@xyz user]# dmsetup remove ddf1_44656c6c202020201028001510281f033832b7a2f6678dab1
[root@xyz user]# dmsetup ls
ddf1_44656c6c202020201028001510281f033832b7a2f6678dab   (253, 0)

[root@xyz user]# dmsetup remove ddf1_44656c6c202020201028001510281f033832b7a2f6678dab

[root@xyz user]# dmsetup ls
No devices found

Mounting using the command “mount -t ext3  /dev/sdb1 /mysql”  after the above steps worked fine.

Fatal error: Allowed memory size of 16777216 bytes exhausted (tried to allocate 41 bytes) in /usr/share/cacti/site/lib/xml.php on line 32

Tags

, , ,

I was trying to setup cacti on my ubuntu desktop to gather metrics from few new database servers we were planning to replace. Installed cacti using the following command

apt-get install cacti

Then the next step was to download and the install the cacti mysql templates from http://code.google.com/p/mysql-cacti-templates/. Thanks to Baron, was able to get this up and running fast using the templates. But had one problem though, while importing the mysql template “cacti_host_template_x_mysql_server_ht_0.8.6i-sver1.1.7.xml” cacti bombed with the below error

Fatal error: Allowed memory size of 16777216 bytes exhausted (tried to allocate 41 bytes) in /usr/share/cacti/site/lib/xml.php on line 32

The fix was to edit “/etc/php5/apache2/php.ini” change the parameter “memory_limit” from 16M to 32M and restart apache using the command “/etc/init.d/apache2 restart“. Now cacti was able to import the template without any issues.

While configuring cacti, by mistake selected apache instead of apache2 during the configuration and was not able to access the cacti ui. Ran the following command to reconfigure cacti.

dpkg-reconfigure cacti

Lock wait timeout on slaves

Tags

, , , , ,

We had a one of our slave servers frequently stop replicating with the “Innodb Lock Wait Timeout” error. The slave IO thread would continue to fetch the binlogs while the slave SQL thread kept stopping with  the above mentioned error. The teams initial inclination was to change the innodb lock wait timeout variable from 50 secs to a higher value. It was a read-only slave. Our expectation was there would be no competing writes. Then we started listing what are the next steps possible and what could be wrong.

  1. There could be a user with “super” privilege in the system that was running updates directly on the slave
  2. A backup script that could be locking the tables out for backup
  3. Increase the “innodb lock wait timeout variable
  4. Enable the innodb lock monitor

While we were working on the system, we noticed that there were few select queries that were running for 300+ secs of the following format.

SELECT * FROM s WHERE (s.id in (select s_id from i where o_id = 79168 AND ( u_id = 79168 OR u_id = 0 )))  ORDER BY s.id DESC limit 15

We wanted to see what was innodb upto since the tables in question were innodb. The “show engine innodb status”  showed us something really interesting.

—TRANSACTION 37ECF6, ACTIVE 1 sec, process no 17368, OS thread id 1263749456 fetching rows, thread declared inside InnoDB 481
mysql tables in use 2, locked 1
2242 lock struct(s), heap size 309232, 883550 row lock(s)
MySQL thread id 1948263, query id 17739683 localhost root preparing
SELECT * FROM s WHERE (s.id in (select s_id from i where o_id = 79168 AND ( u_id = 79168 OR u_id = 0 )))  ORDER BY s.id DESC limit 15

It was a select statement with sub query. The oddity comes from “mysql tables in use 2, locked 1“. Why does a select statement need to lock a particular table ? We replaced that particular query the with below one helped us alleviate the locking issue.

SELECT * FROM  s, i WHERE s.id=i.s_id and i.o_id = 79168 and (i.u_id = 79168 OR i.u_id=0) ORDER BY s.id DESC limit 15

Innodb status output for this query :

—TRANSACTION 37ECF7, ACTIVE 5 sec, process no 17368, OS thread id 1263749456 fetching rows, thread declared inside InnoDB 352
mysql tables in use 2, locked 0
MySQL thread id 1948263, query id 17739687 localhost root Sending data
SELECT * FROM  s, i WHERE s.id=i.s_id and i.o_id = 79168 and (i.u_id = 79168 OR i.u_id=0) ORDER BY s.id DESC limit 15

Once the query was replaced and an index was added to o_id column, we were back to sanity.

Follow

Get every new post delivered to your Inbox.

Join 46 other followers