Define a Log Inspection rule for use in policies

The OSSEC Log Inspection engine is integrated into agents and gives Workload Security the ability to inspect the logs and events generated by the operating system and applications running on the computer. Workload Security ships with a standard set of OSSEC Log Inspection rules that you can assign to computers or policies. You can also create custom rules if there is no existing rule that fits your requirements.

Log Inspection Rules issued by Trend Micro are not editable (although you can duplicate them and then edit them).

Log Inspection Rules that are assigned to one or more computers or that are part of a policy cannot be deleted.

To create Log Inspection rules, perform these basic steps:

• Create a new Log Inspection rule
• Decoders
• Subrules
• Real world examples
• Log Inspection rule severity levels and their recommended use
• strftime() conversion specifiers
• Examine a Log Inspection rule

For an overview of the Log Inspection module, see Analyze logs.

Create a new Log Inspection rule

1. In the Workload Security console, go to Policies > Common Objects > Rules > Log Inspection Rules.
2. Click New > New Log Inspection Rule.
3. On the General tab, enter a name and an optional description for the rule.

4. The Content tab is where you define the rule. The easiest way to define a rule is to select Basic Rule and use the options provided to define the rule. If you need further customization, you can select Custom (XML) to switch to an XML view of the rule that you are defining.

   Any changes you make in the Custom (XML) view is lost if you switch back to the Basic Rule view.

For further assistance in writing your own Log Inspection rules using the XML-based language, consult the OSSEC documentation or contact your support provider.

The following options are available for the Basic Rule template:

• Rule ID: The Rule ID is a unique identifier for the rule. OSSEC defines 100000 - 109999 as the space for user-defined rules. Workload Security will pre-populate the field with a new unique Rule ID.
• Level: Assign a level to the rule. Zero (0) means the rule never logs an event, although other rules that watch for this rule may fire.
• Groups: Assign the rule to one or more comma-separated groups. This can be useful when dependency is used because you can create rules that fire on the firing of a rule, or a rule that belongs to a specific group.
• Rule Description: Description of the rule.

• Pattern Matching: This is the pattern the rule will look for in the logs. The rule is triggered on a match. Pattern matching supports Regular Expressions or simpler String Patterns. The String Pattern pattern type is faster than RegEx but it only supports three special operations:

  • ^ (caret): specifies the beginning of text.
  • $ (dollar sign): specifies the end of text.
  • | (pipe): to create a OR between multiple patterns. For information on the regular expression syntax used by the Log Inspection module, see https://www.ossec.net/docs/syntax/regex.html.

• Dependency: Setting a dependency on another rule causes your rule to only log an event if the rule specified in this area has also triggered.

• Frequency: The number of times the rule has to match within a specific time frame before the rule is triggered.
• Time Frame: The period of time in seconds within which the rule has to trigger a certain number of times (the frequency) to log an event.

The Content tab only appears for Log Inspection rules that you create yourself. Log Inspection rules issued by Trend Micro have a Configuration tab instead that displays the Log Inspection rule's configuration options (if any).

1. On the Files tab, type the full path to the files you want your rule to monitor and specify the type of file it is. Note that the path and file name do not support the glob character.

2. On the Options tab, in the Alert section, select whether this rule triggers an alert in Workload Security.
   Alert Minimum Severity sets the minimum severity level that triggers an Alert for rules made using the Basic Rule or Custom (XML) template.

   The Basic Rule template creates one rule at a time. To write multiple rules in a single template you can use the Custom (XML) template. If you create multiple rules with different Levels within a Custom (XML) template, you can use the Alert Minimum Severity setting to select the minimum severity that triggers an Alert for all of the rules in that template.

3. The Assigned To tab lists the policies and computers that are using this Log Inspection rule. Because you are creating a new rule, it has not been assigned yet.

4. Click OK. The rule is ready to be assigned to policies and computers.

Decoders

A Log Inspection rule consists of a list of files to monitor for changes and a set of conditions to be met for the rule to trigger. When the Log Inspection engine detects a change in a monitored log file, the change is parsed by a decoder. Decoders parse the raw log entry into the following fields:

• log: the message section of the event.
• full_log: the entire event.
• location: where the log came from.
• hostname: hostname of the event source.
• program_name: program name from the syslog header of the event.
• srcip: the source IP address within the event.
• dstip: the destination IP address within the event.
• srcport: the source port number within the event.
• dstport: the destination port number within the event.
• protocol: the protocol within the event.
• action: the action taken within the event.
• srcuser: the originating user within the event.
• dstuser: the destination user within the event.
• id: any ID decoded as the ID from the event.
• status: the decoded status within the event.
• command: the command being called within the event.
• url: the URL within the event.
• data: any additional data extracted from the event.
• systemname: the system name within the event.

Rules examine this decoded data looking for information that matches the conditions defined in the rule.

If the matches are at a sufficiently high severity level, any of the following actions can be taken:

• An alert can be raised. Configurable on the Options tab of the Log Inspection Rule's Properties window.
• The event can be written to syslog. Configurable in the SIEM area on Administration > System Settings > Event Forwarding tab.
• The event can be sent to Workload Security. Configurable in the Log Inspection Syslog Configuration setting on the Policy or Computer Editor > Settings > Event Forwarding tab.

Subrules

A single Log Inspection rule can contain multiple subrules. These subrules can be of two types: atomic or composite. An atomic rule evaluates a single event and a composite rule examines multiple events and can evaluate frequency, repetition, and correlation between events.

Groups

Each rule, or grouping of rules, must be defined within a <group></group> element. The attribute name must contain the rules you want to be a part of this group. In the following example we have indicated that our group contains the syslog and sshd rules:

<group name="syslog,sshd,">
</group>

Notice the trailing comma in the group name. Trailing commas are required if you intend to use the <if_group></if_group> tag to conditionally append another sub-rule to this one.

When a set of Log Inspection rules are sent to an agent, the Log Inspection engine on the agent takes the XML data from each assigned rule and assembles it into what becomes essentially a single long Log Inspection rule. Some group definitions are common to all Log Inspection rules written by Trend Micro. For this reason Trend Micro has included a rule called Default Rules Configuration which defines these groups and which always gets assigned along with any other Trend Micro rules. If you select a rule for assignment and haven't also selected the Default Rules Configuration rule, a notice appears informing you that the rule is assigned automatically. If you create your own Log Inspection rule and assign it to a Computer without assigning any Trend Micro-written rules, you must either copy the content of the Default Rules Configuration rule into your new rule, or also select the Default Rules Configuration rule for assignment to the Computer.

Rules, ID, and Level

A group can contain as many rules as you require. The rules are defined using the <rule></rule> element and must have at least two attributes: id and level. The former is a unique identifier for that signature and the latter is the severity of the alert. In the following example, two rules have been created, each with a different rule ID and level:

<group name="syslog,sshd,">
    <rule id="100120" level="5">
    </rule>
    <rule id="100121" level="6">
    </rule>
</group>

Custom rules must have ID values of 100,000 or greater.

You can define additional subgroups within the parent group using the <group></group> tag. This subgroup can reference any of the groups listed in the following table:

If event auto-tagging is enabled, the event is labeled with the group name. Log Inspection rules provided by Trend Micro make use of a translation table that changes the group to a more user-friendly version. For example, login_denied would appear as Login Denied. Custom rules are listed by their group name as it appears in the rule.

Description

Include a <description></description> tag. The description text appears in the event if the rule is triggered.

<group name="syslog,sshd,">
    <rule id="100120" level="5">
        <group>authentication_success</group>
        <description>SSHD testing authentication success</description>
    </rule>
    <rule id="100121" level="6">
        <description>SSHD rule testing 2</description>
    </rule>
</group>

Decoded As

The <decoded_as></decoded_as> tag instructs the Log Inspection engine to only apply the rule if the specified decoder has decoded the log.

<rule id="100123" level="5">
    <decoded_as>sshd</decoded_as>
    <description>Logging every decoded sshd message</description>
</rule>

To view the available decoders, go to the Log Inspection Rule page and click Decoders. Right-click on 1002791-Default Log Decoders and select Properties. Go the Configuration tab and click View Decoders.

Match

To look for a specific string in a log, use the <match></match>. Here is a Linux sshd failed password log:

Jan 1 12:34:56 linux_server sshd[1231]: Failed password for invalid user jsmith from 192.168.1.123 port 1799 ssh2

Use the <match></match> tag to search for the "password failed" string.

<rule id="100124" level="5">
    <decoded_as>sshd</decoded_as>
    <match>^Failed password</match>
    <description>Failed SSHD password attempt</description>
</rule>

Notice the regex caret ("^") indicating the beginning of a string. Although "Failed password" does not appear at the beginning of the log, the Log Inspection decoder will have broken up the log into sections. See Decoders for more information. One of those sections is log which is the message part of the log as opposed to full_log which is the log in its entirety.

The following table lists supported regex syntax:

Conditional statements

Rule evaluation can be conditional upon other rules having been evaluated as true. The <if_sid></if_sid> tag instructs the Log Inspection engine to only evaluate this subrule if the rule identified in the tag has been evaluated as true. The following example shows three rules: 100123, 100124, and 100125. Rules 100124 and 100125 have been modified to be children of the 100123 rule using the <if_sid></if_sid> tag:

<group name="syslog,sshd,">
    <rule id="100123" level="2">
        <decoded_as>sshd</decoded_as>
        <description>Logging every decoded sshd message</description>
    </rule>
    <rule id="100124" level="7">
        <if_sid>100123</if_sid>
        <match>^Failed password</match>
        <group>authentication_failure</group>
        <description>Failed SSHD password attempt</description>
    </rule>
    <rule id="100125" level="3">
        <if_sid>100123</if_sid>
        <match>^Accepted password</match>
        <group>authentication_success</group>
        <description>Successful SSHD password attempt</description>
    </rule>
</group>

Hierarchy of evaluation

The <if_sid></if_sid> tag essentially creates a hierarchical set of rules. That is, by including an <if_sid></if_sid> tag in a rule, the rule becomes a child of the rule referenced by the <if_sid></if_sid> tag. Before applying any rules to a log, the Log Inspection engine assesses the <if_sid></if_sid> tags and builds a hierarchy of parent and child rules.

The hierarchical parent-child structure can be used to improve the efficiency of your rules. If a parent rule does not evaluate as true, the Log Inspection engine will ignore the children of that parent.

Although the <if_sid></if_sid> tag can be used to refer to subrules within an entirely different Log Inspection rule, you should avoid doing this because it makes the rule very difficult to review later on.

The list of available atomic rule conditional options is shown in the following table:

Use the <if_sid>100125</if_sid> tag to make this rule depend on the 100125 rule. This rule is checked only for sshd messages that already matched the successful login rule.

<rule id="100127" level="10">
    <if_sid>100125</if_sid>
    <time>6 pm - 8:30 am</time>
    <description>Login outside business hours.</description>
    <group>policy_violation</group>
</rule>

Restrictions on the Size of the Log Entry

The following example takes the previous example and adds the maxsize attribute which tells the Log Inspection engine to only evaluate rules that are less than the maxsize number of characters:

<rule id="100127" level="10" maxsize="2000">
    <if_sid>100125</if_sid>
    <time>6 pm - 8:30 am</time> 
    <description>Login outside business hours.</description>
    <group>policy_violation</group>
</rule>

The following table lists possible atomic rule tree-based options:

Composite rules

Atomic rules examine single log entries. To correlate multiple entries, you must use composite rules. Composite rules are supposed to match the current log with those already received. Composite rules require two additional options: the frequency option specifies how many times an event or pattern must occur before the rule generates an alert, and the timeframe option tells the Log Inspection engine how far back, in seconds, it should look for previous logs. All composite rules have the following structure:

<rule id="100130" level="10" frequency="x" timeframe="y">
</rule>

For example, you could create a composite rule that creates a higher severity alert after five failed passwords within a period of 10 minutes. Using the <if_matched_sid></if_matched_sid> tag you can indicate which rule needs to be seen within the desired frequency and timeframe for your new rule to create an alert. In the following example, the frequency attribute is set to trigger when five instances of the event are seen and the timeframe attribute is set to specify the time window as 600 seconds.

The <if_matched_sid></if_matched_sid> tag is used to define which other rule the composite rule will watch:

<rule id="100130" level="10" frequency="5" timeframe="600">
    <if_matched_sid>100124</if_matched_sid>
    <description>5 Failed passwords within 10 minutes</description>
</rule>

There are several additional tags that you can use to create more granular composite rules. These rules, as shown in the following table, allow you to specify that certain parts of the event must be the same. This allows you to tune your composite rules and reduce false positives:

If you wanted your composite rule to alert on every authentication failure, instead of a specific rule ID, you could replace the <if_matched_sid></if_matched_sid> tag with the <if_matched_ group></if_matched_ group> tag. This allows you to specify a category, such as authentication_ failure, to search for authentication failures across your entire infrastructure.

<rule id="100130" level="10" frequency="5" timeframe="600">
    <if_matched_group>authentication_failure</if_matched_group>
    <same_source_ip />
    <description>5 Failed passwords within 10 minutes</description>
</rule>

In addition to <if_matched_sid></if_matched_sid> and <if_matched_group></if_matched_ group> tags, you can also use the <if_matched_regex></if_matched_regex> tag to specify a regular expression to search through logs as they are received.

<rule id="100130" level="10" frequency="5" timeframe="600">
    <if_matched_regex>^Failed password</if_matched_regex>
    <same_source_ip />
    <description>5 Failed passwords within 10 minutes</description>
</rule>

Real world examples

Workload Security includes many default Log Inspection rules for dozens of common and popular applications. Through Security Updates, new rules are added regularly. In spite of the growing list of applications supported by Log Inspection rules, you may find the need to create a custom rule for an unsupported or custom application.

In this section we will walk through the creation of a custom CMS (content management system) hosted on Microsoft Windows Server with IIS and .Net platform, with a Microsoft SQL Server database as the data repository.

The first step is to identify the following application logging attributes:

1. Where does the application log to?
2. Which Log Inspection decoder can be used to decode the log file?
3. What is the general format of a log file message?

For our custom CMS example the answers are as follows:

1. Windows Event Viewer
2. Windows Event Log (eventlog)

3. Windows Event Log Format with the following core attributes:

   • Source: CMS
   • Category: None
   • Event: <Application Event ID>

The second step is to identify the categories of log events by application feature, and then organize the categories into a hierarchy of cascading groups for inspection. Not all inspected groups need to raise events; a match can be used as a conditional statement. For each group, identify the log format attributes which the rule can use as matching criteria. This can also be performed by inspecting all application logs for patterns and logical groupings of log events.

For example, the CMS application supports the following functional features which we will create Log Inspection rules for:

• CMS Application Log (Source: CMS)
  • Authentication (Event: 100 to 119)
    • User Login successful (Event: 100)
    • User Login unsuccessful (Event: 101)
    • Administrator Login successful (Event: 105)
    • Administrator Login unsuccessful (Event: 106)
  • General Errors (Type: Error)
    • Database error (Event: 200 to 205)
    • Runtime error (Event: 206-249)
  • Application Audit (Type: Information)
    • Content
      • New content added (Event: 450 to 459)
      • Existing content modified (Event: 460 to 469)
      • Existing content deleted (Event: 470 to 479)
    • Administration
      • User
        • New User created (Event: 445 to 446)
        • Existing User deleted (Event: 447 to 449)

This structure provides you with a good basis for rule creation. Now to create a new Log Inspection rule in Workload Security.

To create the new CMS Log Inspection Rule:

1. In the Workload Security console, go to Policies > Common Objects > Rules > Log Inspection Rules and click New to display the New Log Inspection Rule Properties window.
2. Give the new rule a name and a description, and then select the Content tab.
3. To start with a basic rule template, select Basic Rule.
4. The Rule ID field is automatically populated with an unused ID number of 100,000 or greater, the IDs reserved for custom rules.
5. Set Level to Low (0).
6. Give the rule an appropriate Group name. In this case, cms.

7. Provide a short rule description.

   

8. Select the Custom (XML) option. The options you selected for your Basic rule are converted to XML.

   

9. Select the Files tab and click Add File to add any application log files and log types to which to apply the rule. In this case, Application and eventlog as the file type.

   

   Eventlog is a unique file type in Workload Security because the location and filename of the log files do not have to be specified. Instead, it is sufficient to type the log name as it is displayed in the Windows Event Viewer. Other log names for the eventlog file type might be Security, System, Internet Explorer, or any other section listed in the Windows Event Viewer. Other file types require the log file's location and filename. C/C++ strftime() conversion specifiers are available for matching on filenames. See the following table for a list of some of the more useful ones.

10. Click OK to save the basic rule.

11. Working with the basic rule Custom (XML) created, begin adding new rules to the group based on the log groupings identified previously. Set the base rule criteria to the initial rule. In the following example, the CMS base rule has identified Windows Event Logs with a Source attribute of CMS:

    <group name="cms">
        <rule id="100000" level="0">
        <category>windows</category>
        <extra_data>^CMS</extra_data>
        <description>Windows events from source 'CMS' group messages.</description>
    </rule>

12. Build up subsequent rules from the identified log groups. The following example identifies the authentication and login success and failure and logs by Event IDs.

    <rule id="100001" level="0">
        <if_sid>100000</if_sid>
        <id>^100|^101|^102|^103|^104|^105|^106|^107|^108|^109|^110</id>
        <group>authentication</group>
        <description>CMS Authentication event.</description>
    </rule>
    
    <rule id="100002" level="0">
        <if_group>authentication</if_group>
        <id>100</id>
        <description>CMS User Login success event.</description>
    </rule>
    
    <rule id="100003" level="4">
        <if_group>authentication</if_group>
        <id>101</id> <group>authentication_failure</group>
        <description>CMS User Login failure event.</description>
    </rule>
    
    <rule id="100004" level="0">
        <if_group>authentication</if_group>
        <id>105</id>
        <description>CMS Administrator Login success event.</description>
    </rule>
    
    <rule id="100005" level="4">
        <if_group>authentication</if_group>
        <id>106</id>
        <group>authentication_failure</group>
        <description>CMS Administrator Login failure event.</description>
    </rule>

13. Add any composite or correlation rules using the established rules. The following example shows a high severity composite rule that is applied to instances where the repeated login failures have occurred 5 times within a 10 second time period:

    <rule id="100006" level="10" frequency="5" timeframe="10">
        <if_matched_group>authentication_failure</if_matched_group>
        <description>CMS Repeated Authentication Login failure event.</description>
    </rule>

14. Review all rules for appropriate severity levels. For example, error logs should have a severity of level 5 or higher. Informational rules would have a lower severity.

15. Finally, open the newly created rule, select the Configuration tab and copy your custom rule XML into the rule field. Click Apply or OK to save the change.

Once the rule is assigned to a policy or computer, the Log Inspection engine should begin inspecting the designated log file immediately.

The complete Custom CMS Log Inspection rule:

<group name="cms">
    <rule id="100000" level="0">
        <category>windows</category>
        <extra_data>^CMS</extra_data>
        <description>Windows events from source 'CMS' group messages.</description>
    </rule>
    <rule id="100001" level="0">
        <if_sid>100000</if_sid>
        <id>^100|^101|^102|^103|^104|^105|^106|^107|^108|^109|^110</id>
        <group>authentication</group>
        <description>CMS Authentication event.</description>
    </rule>
    <rule id="100002" level="0">
        <if_group>authentication</if_group>
        <id>100</id>
        <description>CMS User Login success event.</description>
    </rule>
    <rule id="100003" level="4">
        <if_group>authentication</if_group>
        <id>101</id>
        <group>authentication_failure</group>
        <description>CMS User Login failure event.</description>
    </rule>
    <rule id="100004" level="0">
        <if_group>authentication</if_group>
        <id>105</id>
        <description>CMS Administrator Login success event.</description>
    </rule>
    <rule id="100005" level="4">
        <if_group>authentication</if_group>
        <id>106</id>
        <group>authentication_failure</group>
        <description>CMS Administrator Login failure event.</description>
    </rule>
    <rule id="100006" level="10" frequency="5" timeframe="10">
        <if_matched_group>authentication_failure</if_matched_group>
        <description>CMS Repeated Authentication Login failure event.</description>
    </rule>
    <rule id="100007" level="5">
        <if_sid>100000</if_sid>
        <status>^ERROR</status>
        <description>CMS General error event.</description>
        <group>cms_error</group>
    </rule>
    <rule id="100008" level="10">
        <if_group>cms_error</if_group>
        <id>^200|^201|^202|^203|^204|^205</id>
        <description>CMS Database error event.</description>
    </rule>
    <rule id="100009" level="10">
        <if_group>cms_error</if_group>
        <id>^206|^207|^208|^209|^230|^231|^232|^233|^234|^235|^236|^237|^238|
        ^239^|240|^241|^242|^243|^244|^245|^246|^247|^248|^249</id>
        <description>CMS Runtime error event.</description>
    </rule>
    <rule id="100010" level="0">
        <if_sid>100000</if_sid>
        <status>^INFORMATION</status>
        <description>CMS General informational event.</description>
        <group>cms_information</group>
    </rule>
    <rule id="100011" level="5">
        <if_group>cms_information</if_group> 
        <id>^450|^451|^452|^453|^454|^455|^456|^457|^458|^459</id>
        <description>CMS New Content added event.</description>
    </rule>
    <rule id="100012" level="5">
        <if_group>cms_information</if_group>
        <id>^460|^461|^462|^463|^464|^465|^466|^467|^468|^469</id>
        <description>CMS Existing Content modified event.</description>
    </rule>
    <rule id="100013" level="5">
        <if_group>cms_information</if_group>
        <id>^470|^471|^472|^473|^474|^475|^476|^477|^478|^479</id>
        <description>CMS Existing Content deleted event.</description>
    </rule>
    <rule id="100014" level="5">
        <if_group>cms_information</if_group>
        <id>^445|^446</id>
        <description>CMS User created event.</description>
    </rule>
    <rule id="100015" level="5">
        <if_group>cms_information</if_group>
        <id>^447|449</id>
        <description>CMS User deleted event.</description>
    </rule>
</group>  

Log Inspection rule severity levels and use

strftime() conversion specifiers

For more information, see the following:

• https://www.php.net/manual/en/function.strftime.php
• http://www.cplusplus.com/reference/ctime/strftime/

Examine a Log Inspection rule

Log Inspection rules are found in the Workload Security console at Policies > Common Objects > Rules > Log Inspection Rules.

Log Inspection rule structure and the event matching process

The following illustration displays the contents of the Configuration tab of the Properties window of the Microsoft Exchange Log Inspection rule:

The following is the structure of the rule:

• 3800 - Grouping of Exchange Rules - Ignore
  • 3801 - Email rcpt is not valid (invalid account) - Medium (4)
    • 3851 - Multiple email attempts to an invalid account - High (9)
      • Frequency - 10
      • Time Frame - 120
      • Ignore - 120
  • 3802 - Email 500 error code - Medium (4)
    • 3852 - Email 500 error code (spam) - High (9)
      • Frequency - 12
      • Time Frame - 120
      • Ignore - 240

The Log Inspection engine applies log events to this structure and see if a match occurs. For example, if an Exchange event occurs, and this event is an email receipt to an invalid account, the event matched line 3800 (because it is an Exchange event). The event is then be applied to line 3800's sub-rules: 3801 and 3802.

If there is no further match, this cascade of matches stops at 3800. Because 3800 has a severity level of Ignore, no Log Inspection event would be recorded.

However, an email receipt to an invalid account does match one of 3800's subrules: subrule 3801. Subrule 3801 has a severity level of Medium(4). If the matching stopped here, a Log Inspection event with a severity level of Medium(4) would be recorded.

But there is still another subrule to be applied to the event: subrule 3851. Subrule 3851 with its three attributes matches if the same event has occurred 10 times within the last 120 seconds. If so, a Log Inspection event with a severity High(9) is recorded. The Ignore attribute tells sub-rule 3851 to ignore individual events that match sub-rule 3801 for the next 120 seconds. This is useful for reducing noise.

Assuming the parameters of subrule 3851 have been matched, a Log Inspection event with Severity High(9) is now recorded.

Looking at the Options tab of the Microsoft Exchange Rule, it is reasonable to assume that Workload Security will raise an alert if any subrules with a severity level of Medium(4) have been matched. Since this is the case in the example, the alert is raised if Alert when this rule logs an event is selected.

Duplicate Subrules

Some Log Inspection rules have duplicate subrules. To see an example, open the Microsoft Windows Events rule and select the Configuration tab. Note that subrule 18125 (Remote access login failure) appears under subrules 18102 and 18103. Also note that in both cases subrule 18125 does not have a severity value, it only says See Below.

Instead of being listed twice, Rule 18125 is listed once at the bottom of the Configuration page: