Edit history

1. 10/9/07, added section that describes how to activate "Delete files slowly".
2. 10/25/07, added section titled "Check out the mythtv wiki". Contains additional tweaks that helped solve problems that started to re-appear after upgrading from R5D1 to R5F27.
3. 06/29/08, added note in section "Increase buffer size for IVTV"

Fixing IOBOUND & similiar problems

This page describes how to resolve a series of problems whose symptoms includes:

1. Error messages in /var/log/mythtv/mythbackend.log that containing the words "IOBOUND" or "Cannot allocate memory"
2. Error messages in /var/log/kern.log that containing the words "application not reading fast enough" or "out of vmalloc space"
3. Glitchy recordings, if those recordings are captured at the same time that a subset of these problems occur.

I used to see the "IOBOUND" and "**application not reading fast enough**" errors multiple times/day. I made the changes outlined below in phases. Each change reduced the frequency of problems, but it required all of them to fully solve the problem. With all of these changes in place I have not had any of these problems in several months.

Credits

Thanks and credit go to Cecil, TJC, rando, sgunther, Brendan & bruce_s01 whose threads provided significant input into what you see here. You can see their original posts at

1. http://mysettopbox.tv/phpBB2/viewtopic.php?t=1043
2. http://mysettopbox.tv/phpBB2/viewtopic.php?t=13534&highlight=encoder+mpeg+stream+buffers+full
3. http://mysettopbox.tv/phpBB2/viewtopic.php?t=14097&postdays=0&postorder=asc&highlight=cause+application+reading+fast+enough&start=0
4. http://mysettopbox.tv/phpBB2/viewtopic.php?t=11875&highlight=buffer+vmalloc

Optimize database to improve performance

Over time database performance will degrade and contribute to "IOBOUND" problems and sluggish responses to various requests. To restore full performance & minimize the database's physical size, do the following periodically:

1. Make a database backup using mythbackup.
2. /etc/init.d/mythtv-backend stop
3. /etc/init.d/mysql stop
4. Disable any watchdog scripts you may be running.
5. cd /var/lib/mysql/mythconverg
6. myisamchk -r *.MYI
7. myisamchk -a *.MYI
8. /etc/init.d/mysql start
9. /etc/init.d/mythtv-backend start

You can also download and use the following script: optimize_db.sh which automates the process.

Reduce the volume of program guide data

Reducing the number of days of program guide data that I kept in my database from the default of 14 to 9 also reduced the incidence of "IOBOUND" errors for me. One way to determine if this change will benefit you is to look in the file /var/log/mythtv/mythbackend.log for a message that contains the following text "Scheduled nnn items in", where "nnn" is in integer value. The higher the value of "nnn", the higher the probability that your system will become IOBOUND when the scheduler runs. Before I made this change "nnn" was typically in the range of 680 - 750 for me; after making this change "nnn" is is in the range of 430 - 485.

Since you already have 14 days of program guide data in your database, it will take 5 days before you see the full benefit of this change:

1. On R5D1,R5F27: utilities/setup->setup->general->"mythfilldatabase" (screen 7), modify the field "mythfilldatabase arguments" to read
   • "--quiet --max-days 9", without the quotes.

If you have any unused video sources hanging around, delete them using the myth-setup utility. This reduces the volume of data downloaded from zap2it into the database. Only do this if the video source is not needed and is not connected to any tuner cards:

1. Make a database backup using mythbackup.
2. ALT-S
3. Select option "3 video-sources"
4. Highlight the unused video source
5. "M" and choose the "delete..." option

Increase buffer size for IVTV

The following steps eliminated the "application not reading fast enough" errors for me. This procedure increases IVTV's buffer sizes from the default of 4/8/4/4 to 8/32/8/8. Some have reported that the problem goes away with a smaller increase to 4/16/4/4 - you can experiment to see what works best for you. You need enough RAM for this to be OK - I have 1 GB

Note from joelsplace: I used options ivtv enc_yuv_buffers=8 enc_mpg_buffers=32 enc_vbi_buffers=8 enc_pcm_buffers=320 (pcm default R5F27) because pcm is in KB so the below line #4 changed mine to 8KB not 8MB.

1. Note: Sometimes the option names for ivtv change from release to release. To find all of the available options, simply enter
   • modinfo ivtv | more
2. R5D1: Add the following line to the end of /etc/modprobe.conf:
   • options ivtv yuv_buffers=8 mpg_buffers=32 vbi_buffers=8 pcm_buffers=8
3. R5E50: Add the following line after the existing line in "/etc/mythtv/modules/ivtv" that reads "options msp3400 once=1":
   • options ivtv yuv_buffers=8 mpg_buffers=32 vbi_buffers=8 pcm_buffers=8
4. R5F27: Add the following line after the existing line in "/etc/mythtv/modules/ivtv" that reads "options msp3400 once=1":
   • options ivtv enc_yuv_buffers=8 enc_mpg_buffers=32 enc_vbi_buffers=8 enc_pcm_buffers=8
5. All versions: Reboot and verify that you are now allocating 32MB to your mpg buffers. You can do this by looking for the following line in /var/log/kern.log:
   • ivtv0: Allocate DMA encoder MPEG stream: 1024 x 32768 buffers (32768KB total)

Delete files slowly

Your system can delete old recordings at any time, depending on how your auto-expiration and other settings are configured. The deletion of recordings, especially hi-def recordings, can cause a lot of I/O and contribute to the problems outlined at the start of this article. To reduce the odds of these deletions causing problems, turn on the "Delete files slowly" option that is available in R5F27:

1. Exit the mythtv front end.
2. Halt your backend (ALT-2)
3. Bring up the mythtv setup utility (ALT-s)
4. Select "1-General"
5. The second screen ("Host-specific Backend Setup") contains an option named "Delete files slowly". Select this option (ie, there should be a check-mark on this option.)
6. Keep hitting "Next" until you get back to the main screen that starts with "1-General".
7. Exit the setup utility ("esc"). Don't worry about any messages it may give you about running mythfilldatabase.
8. Restart the backend (ALT-3)
9. Restart the frontend (ALT-M)

I found this helped my system even though I already use XFS, which is already very efficient at large file deletes.

Fixing problematic watchdog scripts

There are a lot of "watchdog" scripts scattered around various sites that are designed to detect backend failures and execute an automatic restart. Some of these scripts include a call to the status port (6544) to see if the backend is still running. This can be a problem as a call to the status port will issue ~500 database queries. A better approach that works under R5D1 & R5F27 is to invoke the "settings" screen, which only issues ~10 database queries:

1. Look in your script for a command that reads something like
   • lynx -dump http://localhost:6544
2. Replace that command with
   • lynx -dump http://localhost/mythweb/settings
3. The script may grep an output file for a specific string such as "Schedule". If your script does this, replace the search string with the following: "This is the index page"

Switch your /myth partition to XFS

The default file system format for the /myth partition is "ext3". While this file system is very stable, deleting large file requires significant disk I/O activity and will increase the odds of experiencing IOBOUND problems. Instructions for switching to XFS can be found here: Filesystem switch

Do not switch your root partition ("/") to XFS, as your mysql database is located on the root partition. I have read articles indicating that database corruption can occur if a mysql database is placed on an XFS partition.

More recent releases of mythtv provide an option to "delete files slowly". I have not tried this option, but it may be the case that turning this option on will help if you would rather not convert to XFS. This option is found in the mythtv-setup program (hit ALT-S to bring this up) in the general section. On the second tab, there is a check box you can select labeled "delete files slowly".

Check out the mythtv wiki

Additional performance enhancements are documented in the mythtv wiki here. After upgrading from R5D1 to R5F27 I started getting IOBOUND errors twice/week. Applying the following tweaks documented in that article has helped significantly:

1. Enable fast writes in the Nvidia driver
   • R5F27: Add the following line to the end of /etc/modprobe.d/nvidia-kernel-nkc: options nvidia NVreg_EnableAGPFW=1
2. Enable async mode in NFS by adding the "async" option
   • R5F27: As an example, my "/myth" entry in /etc/exports new reads: /myth *(rw,async)
3. Increase the default allocation size in the XFS file system to reduce fragmentation. The article suggests increasing the default allocation size to 512m, but I only increased it to 512k so as to avoid wasting too much disk space with smaller files.
   • R5F27: As an example, one of my mount points in /etc/fstab now reads: /dev/sda1 /myth xfs defaults,auto,allocsize=512k 0 2
4. Modify the mysql settings as suggested. Note: Only make the subset of changes that increase buffer/cache sizes.
   • R5F27: These changes are made in /etc/mysql/my.conf

I/O intensive background jobs

If you have scripts that run periodically in the background, check carefully to see how much I/O they do. A process that is I/O intensive but not CPU intensive will cause IOBOUND problems even if you "nice" it. As an example, I had a script that ran every 20 minutes in "nice" mode. Upon close examination I found that this script was triggering copies of ~100MB of data on each run, most of which was unnecessary. I re-wrote it and it now drives under 100KB of I/O on each call.

Increase vmalloc space

There are releases where there is not enough memory reserved for vmalloc. If you see messages in kern.log that contain the words "**out of vmalloc space**" or messages in mythbackend.log that contain the words "Cannot allocate memory", then you have this problem. If you are not seeing either of those specific error messages, then I would not make this change unless you have tried everything else in this article and are still having the other problems identified in this article. These instructions will work for R5D1 & R5F27:

1. Edit the file /etc/lilo.conf
2. Add the following option to the end of the append line: vmalloc=256m
3. Run: lilo -v
4. reboot