1679938 - DiskFullEvent on Log Volume

Symptom

You notice, that the log volume is full and the database does not accept any new requests.

Additionally, the trace of the service with the full log volume contains DiskFullEvent messages:

LoggerImpl.cpp : Logger notified of new DiskFull: DiskFullEvent[id= XXX, path= /hana/log/<SID>/mntXXXXX/hdbXXXXX/, state= NEW]

Other Terms

HANA, in-Memory, logvolume, full, no requests, DISKFULL event

Reason and Prerequisites

Reason:

Due to an architectural limitation, it was not possible to successfully perform a log backup when no space in the log area was available.

Affected Releases:

• SAP HANA 1:
  • All Revisions
• SAP HANA 2:
  • Revisions of SPS00 - SPS05

Prerequisites:

• The log volume is full and the database does not respond or start anymore.

Solution

Apply one of the following SAP HANA Revisions:

• SAP HANA 2:
  • Revisions >= 060.00 (SPS06)
  • or higher
• or higher

Workaround:

1. Make sure that you are logged in as <SID>adm user on the host on which you experience the log volume full situation.
2. Stop the database:

   /usr/sap/hostctrl/exe/sapcontrol -nr <instance_number> -function StopSystem HDB

3. Wait that the database on all hosts is stopped completely.
4. Change directory into folder mnt00001 of the log volume (Default: /hana/log/<SID>/), e.g.:

   cd /hana/log/<SID>/mnt00001

5. You have to move one of the log volumes temporarily to another volume where enough space is available.
   You should free at least 2 GB of space to ensure that the database has enough space to start. To find out the space consumption of each volume execute:

   du -sh *

6. Move a volume which consumes at least 2 GB of space (e.g. hdb00003) to a volume with sufficient free space, e.g. to the data volume (Default: /hana/data/<SID>), e.g.:

   mkdir /hana/data/<SID>/logs_temporary

   mv hdb00003 /hana/data/<SID>/logs_temporary

7. Create a symbolic link to the new folder in the old location:

   ln -s /hana/data/<SID>/logs_temporary/hdb00003 /hana/log/<SID>/mnt00001/hdb00003

8. Start the database:

   /usr/sap/hostctrl/exe/sapcontrol -nr <instance_number> -function StartSystem HDB

9. Wait until log backups are performed.
10. Use the following SQL-Statement on SYSTEMDB and tenant level to clean up the logvolume:

    ALTER SYSTEM RECLAIM LOG;

11. Stop the database again:

    /usr/sap/hostctrl/exe/sapcontrol -nr <instance_number> -function StopSystem HDB

12. Wait that the database on all hosts is stopped completely.

13. Remove the symbolic link:

    rm -f /hana/log/<SID>/mnt00001/hdb00003

14. Move the log volume back to its original location:

    mv /hana/data/<SID>/logs_temporary/hdb00003 /hana/log/<SID>/mnt00001

15. Remove the temporary log folder:
    
    rmdir /hana/data/<SID>/logs_temporary

16. Start the database

    /usr/sap/hostctrl/exe/sapcontrol -nr <instance_number> -function StartSystem HDB

Remarks:

If you have not noticed a DiskFullEvent, but your service is unresponsive, you might encounter a limit of the number of log segments per log partition. For further details please see SAP Note 2072410.