Tags

,

Every hour, we saw the diskspace on our root partition go up from 45 %  to 90 % and come back to 45 %. Initially thought mysql was processing some temp tables and in the next hour we kept watching the temp location. In terms of files / diskspace usage using “du / ls” nothing changed. But lsof provided an interesting output.

Output 1 :

mysqld    31719   mysql    6u   REG                8,1          0   149240 /var/tmp/ibOLLuRa (deleted)
mysqld    31719   mysql    7u   REG                8,1        102   149243 /var/tmp/ibNuSvI9 (deleted)
mysqld    31719   mysql    8u   REG                8,1          0   149245 /var/tmp/ibqq2wz8 (deleted)
mysqld    31719   mysql    9u   REG                8,1          0   149255 /var/tmp/ibVN66Rh (deleted)
mysqld    31719   mysql   13u   REG                8,1          0   149256 /var/tmp/ibslSJJA (deleted)
mysqld    31719   mysql   15u  unix 0x000001085e445680            19044143 /tmp/mysql.sock
mysqld    31719   mysql   18u   REG                8,1 1128267776   146656 /var/tmp/MLHbDVOq (deleted)

Output 2 :

mysqld    31719   mysql    6u   REG                8,1          0   149240 /var/tmp/ibOLLuRa (deleted)
mysqld    31719   mysql    7u   REG                8,1        102   149243 /var/tmp/ibNuSvI9 (deleted)
mysqld    31719   mysql    8u   REG                8,1          0   149245 /var/tmp/ibqq2wz8 (deleted)
mysqld    31719   mysql    9u   REG                8,1          0   149255 /var/tmp/ibVN66Rh (deleted)
mysqld    31719   mysql   13u   REG                8,1          0   149256 /var/tmp/ibslSJJA (deleted)
mysqld    31719   mysql   15u  unix 0x000001085e445680            19044143 /tmp/mysql.sock
mysqld    31719   mysql   18u   REG                8,1 1396703232   146656 /var/tmp/MLHbDVOq (deleted)

The size occupied by a file marked deleted kept increasing :( Strange but it seems to be happening on every dataload. The size change in this file corroborates perfectly with the space usage reported by “df” but not with “du”.

We are setting up the tmp location to a filesystem with more space for now before we investigate further.

About these ads