8/14/2012LogRhythm 6.1 MPE Rule Builder Cheat Sheet Parsing Fields and Tags The following table provides a list of all the meta-data fields LogRhythm can parse and their associated parsing tag(s). Also provided is the regular expression embedded as part of the tag. This regular expression can be overridden, an explanation of how this done follows the table. *** All Mapping and Parsing tags are lower case *** Field Origin Host Impacted Host Origin Port Impacted Port Origin MAC Address Description The host from which activity originated (i.e., attacker) The host which was effected by the activity (i.e., target) The port from which activity originated (i.e., client port). The port to which activity is targeted (i.e., server port) The MAC Address which activity originated. Tag(s) <sip> IP Address Default Regex <sname> Host Name (?<sipa>1??\d{1,2}|2[0-4]\d|25[0-5])\. (?<sipb>1??\d{1,2}|2[0-4]\d|25[0-5])\. (?<sipc>1??\d{1,2}|2[0-4]\d|25[0-5])\. (?<sipd>1??\d{1,2}|2[0-4]\d|25[0-5]) ([^\s\.]+\.?)+ <sipn> IP or Host Name (<sip>|<sname>) <snatip> NAT IP Address <dip> IP Address Same as SIP and DIP tags <dname> Host Name (?<dipa>1??\d{1,2}|2[0-4]\d|25[0-5])\. (?<dipb>1??\d{1,2}|2[0-4]\d|25[0-5])\. (?<dipc>1??\d{1,2}|2[0-4]\d|25[0-5])\. (?<dipd>1??\d{1,2}|2[0-4]\d|25[0-5]) ([^\s\.]+\.?)+ <dipn> IP or Host Name (<dip>|<dname>) <dnatip> NAT IP Address <sport> Source Port Same as SIP and DIP tags <snatport> NAT Port \d+ <dport> Destination Port \d+ <snatport> NAT Port \d+ <smac> (\w{2}(:|-)?){6} \d+ + Page 2 of 11 . file) referenced or impacted by activity reported in the log. <protnum> use for IANA 1??\d{1. The Windows or DNS domain name referenced or impacted by activity reported in the log. Inc The MAC Address which was effected by the activity.MPE Rule Builder Cheat Sheet Field Description Tag(s) Default Regex <dmac> (\w{2}(:|-)?){6} Origin Interface <sinterface> \w+ Impacted Interface <dinterface> \w+ Impacted MAC Address Protocol Login Account Group Domain Object URL Copyright 2010 LogRhythm.e.e.2}|2[0-4]\d|25[0-5] Protocol Number <protname> use for standard protocol abbreviations/names \w+ <login> \w+ <account> \w+ <group> \w+ <domain> \w+ <object> \w+ <objectname> \w+ <url> https?://.. The user account impacted by activity reported in the log. The group or role impacted by activity reported in the log.. The network protocol used for the activity (i. The URL referenced or impacted by activity reported in the log. TCP) The user associated with the activity reported in the log. The resource (i. User. The sender of an email or called from number for a VOIP log. system. The subject of an email. Inc Page 3 of 11 . or application session System or application process Tag(s) Default Regex <vmid> \w+ <sender> [^\s]+@[^\s]+ <recipient> [^\s]+@[^\s]+ <subject> \w+ <session> \w+ <process> \w+ <processid> \d+ Severity <severity> \w+ Version <version> \w+ Command <command> \w+ Sender Recipient Subject Session Process Copyright 2010 LogRhythm. The recipient of an email or called to number for a VOIP log.MPE Rule Builder Cheat Sheet Field Description Vendor Message ID The specific vendor log/event identifier for the log. <itemsout> Duration The duration of a session. use the following tags. Default Regex [0123456789\. job. etc. <gigabyteout> <terabitsin>. <kilobytesout> <megabitsin>. <terabytesout> <petabitsin>. <bytesout> <kilobitsin>. Inc [0123456789\. <megabyteout> <gigabitsin>.]+ the data represents packet counts <packetsin>. system. or process Tag(s) Use the appropriate tag based upon the units represented by the log data. <bitsout> <bytesin>. <itemsin>. <gigabitsout> <gigabytein>. <kilobitsout> <kilobytesin>. If the log contains a specific start and end date/time. <petabytesout> Use the following tags if [0123456789\. <timestart>. Copyright 2010 LogRhythm.]+ Page 4 of 11 . activity. <packetsout> Use the following tags if the data represents anything else. system. <terabitsout> <terabytesin>. or process Items (ie packets) sent/received from a device.MPE Rule Builder Cheat Sheet Field Bytes In/Out Items In/Out Description Bytes sent/received from a device. <petabitsout> <petabytesin>. <megabitsout> <megabytein>.]+ <bitsin>. ]+ <quantity> [0123456789\.]+ <microseconds> [0123456789\. use the following tags to capture the time field elements.MPE Rule Builder Cheat Sheet Field Description Tag(s) <timeend> Default Regex [0123456789\.]+ If the log contains a value representing a time.]+ <rate> [0123456789\.]+ <amount> [0123456789\.]+ <minutes> [0123456789\.]+ <nanoseconds> [0123456789\.]+ Page 5 of 11 . Inc The size of something The quantity of something The amount of something The rate of something <days> [0123456789\. Size Quantity Amount Rate Copyright 2010 LogRhythm.]+ <hours> [0123456789\.]+ <milliseconds> [0123456789\.]+ <seconds> [0123456789\.]+ <size> [0123456789\. * . or Is not the optimal regex from a performance perspective …the default should be overridden.?\w*) opened (?<object>\w+\. the following syntax should be used. Regular Expression Characters & Practices Following is an overview of regular expression characters and recommend practices.* . There sole purpose is to identify portions of the log message that should be used in the development of sub-rules.* . (?<[tagname]>[regex]) For example. To override the default regex. the default regex’s should be overridden and the base-rule should be: User (?<login>\w+\.blow opened AnnualReport. This is due to the fact that a period is not a word character and the default regex of “\w+” would only match up to the period. Instead.* . suppose your regex needs to match file names with a specific extension such as the sample log message below: User joe.pdf) Now. Tag Field Type <tag1> <tag2> <tag3> <tag4> <tag5> Text Text Text Text Text Default Regex .MPE Rule Builder Cheat Sheet Mapping Tags 5 additional tags are available for identifying data in the log specifically for sub-rules. These tags do not parse text into meta-data fields. Inc Characters Matched Example Page 6 of 11 . the base-rule will parse anything for login starting with a word character that optionally contains a period followed be additional word characters. Match Characters Notation Copyright 2010 LogRhythm.* Overriding the Default Regex If the default regex for a parsing tag… • • Will not properly parse the correct data out of the log message.pdf If the base-rule was written as: User <login> opened <object> The value parsed for login would joe and the value for object would be AnnualReport. not the start of a word.9}) Match the previous item 0 or 1 times Match the previous item 1 or more times Match the previous item 0 or more times Example \w{4} matches AAAA but not A \w{4. essentially the point between a word character (a-z. carriage return. 0-9 and the underscore character _ Any non-word character Any whitespace character. matches anything [abc] matches a or b or c but no other character [^abc] matches def but not abc Repetition Characters Notation {n} {n.3} matches AA and AAA but not A or AAAA A? matches A or nothing but not AA A+ matches A. or if it is a multi-line string then at the end of a line. A-Z. the multi-line flag is ignored The preceding pattern must be at the end of the string. A or any number of A’s Positional Characters Notation ^ $ \A \Z \b Description The following pattern must be at the start of the string. AA. The start of a word. A-Z. \B Grouping Notation ()? ()+ Characters Matched Example Matches the pattern inside the brackets 0 (Error)? Matches Error or nothing or 1 times Matches the pattern inside the brackets 1 (\w+\s)+ Matches AA AA Copyright 2010 LogRhythm. AAA but not nothing A* matches nothing. tab newline. The preceding pattern must be at the end of the string. } {n. Matches anything but a space Any character Any character between the brackets \w \W \s \S . The preceding pattern must be at the start of the string. This matches a position that is not a word boundary. For multi-line text (string containing a carriage return) the multi-line flag option needs to be set. a-z. form feed and vertical tab. [] [^ ] Matches any character except the characters appearing after the ^ and before the]. or if it’s a multi-line string. _) and a non-word character. \d\d\d matches 101 but not 10a \D\D\D matches abc but not 101 \w\w\w matches abc but not ab$ \W\W\W matches $#! but not abc \s\s\s matches but not abc \S\S\S matches a1_ but not .m} ? + * Characters Matched Matches n of the previous item Matches n or more of the previous item Matches at least n and at most m of the previous item if n is 0 that makes the character optional ({. 0-9. at the beginning of a line. the multi-line pattern is ignored The matches a word boundary. } matches AAAAAA but not A A{2.MPE Rule Builder Cheat Sheet \d \D Any digit from 0 to 9 Any character that is not a numeric digit (0 to 9) Any word character. Inc Page 7 of 11 . In order to make the regex take the first occurrence of the next match you use the non-greedy qualifier. The non-greedy qualifier improves performance when you want to match any text value up to a specific text value where the specific text value can be uniquely specified within the regex. incorrect values will be parsed for the login field due to the fact that user occurs twice in the log message.*” will match everything to the end of the log message.*) as soon as “user” is encountered. When the regex engine reaches the end of the log message it will begin looking backwards in the log message for the next match.*user (?<login>\w+\.MPE Rule Builder Cheat Sheet ()* or more times Matches the pattern inside the brackets 0 (\w+\s)* Matches nothing or AA AA or more times Non-Greedy Qualifier (?) The non-greedy qualifier is a question mark (?) following a repetition character (?*+).?\w*) Copyright 2010 LogRhythm. Using this regex will cause “account” parsed into the login field.?\w*) This is because “.*?user (?<login>\w+\.*MsgID=1590. ^. it will use that match and continue.*?MsgID=1590. As soon as it finds the last occurrence of “user” it will match for that portion of the log message. ^. The following regex will parse the correct value into the login field because it will stop the previous match (. For example. The non-greedy qualifier is used to tell the regex engine that it should stop matching the current match as soon as the next match criterion is met. Since the specified regex for “login” will match account.blow user account locked out If you use the following regex. Inc Page 8 of 11 . The non-greedy qualifier is used in combination with a repetition qualifier in order to create a nongreedy match. suppose your regex needs to match the following log: 02/28/2007 16:55:22 MsgID=1590 : Failed authentication for user joe. \d+\. Inc Page 9 of 11 . for example \cV matches Ctrl-V Copyright 2010 LogRhythm.MPE Rule Builder Cheat Sheet Reserved Characters The regex engine used by LogRhythm has 12 reserved characters that have special meaning.\d+ As you can see each of the periods of the IP address are escaped meaning the regex engine will look for the actual period (. otherwise known as the escape character. • The vertical bar or pipe symbol | • The question mark ? • The asterisk or star * • The plus sign + • The opening squiggly bracket { • The closing squiggly bracket } As a simple example of how to escape reserved characters refer to the following regex which is meant to match any IP address: \d+\. Other Special Characters Other special characters which match special cases that cannot be typed into a regular expression: Special Character Description \n Matches newline \r Matches carriage return \t Matches tab \v Matches vertical tab \f Matches form feed \nnn Matches ASCII character specified by octal number nnn Ex: \103 matches C \xnn Matches ASCII character specified by hexadecimal number nn Ex: \x43 matches C \unnnn Matches the Unicode character specified by the four hexadecimal digits replaced by nnnn \cV Matches a control character.) character in the string instead of looking for any character. The reserved characters are: • The opening square brackets [ • The opening round bracket ( • The closing round bracket ) • The backslash \ • The caret ^ • The dollar sign $ • The period . which is the reserved periods special meaning. If any of these characters need to be used as a literal character they will need to be escaped using the backslash (\) character.\d+\. *?user (?<login>\w+\.?\w*) Non-greedy Match . N/A Always match as much constant text as possible.*?user (?<login>\w+\.MPE Rule Builder Cheat Sheet Regex Recommended Practices The following are some recommended practices for regex development.* which will match everything to the end of the log. The more information the regex has to evaluate. Example: ^. The “?” tells the engine to perform a “nongreedy match”.*?match contains this phrase)^.*? Description This is the best way to start any regex if you want to match any characters until a specific set of characters appear. The “^” tells the regex engine to start from the beginning. 02/28/2007 16:55:22 MsgID=1590 : Failed authentication for user joe. Example: ^. the faster it will be at identifying non-matching logs. Example: ^.*?MsgID=1590. For any parsed field.*?MsgID=1590. The default regex for map tags is .*?MsgID=1590.blow user account locked out Name Starting pattern Recommended Pattern ^. Inc Page 10 of 11 .?\w*) Overloading Map Tags (?<[map tag]>[regex]) Map Tags should almost always be overloaded.*?MsgID=1590.?\w*)\s(?<tag1>.*?user (?<login>\w+\.*? If you need to match any characters until a specific set of characters appears. All regex examples use the following log.?\w*)\s(?<tag1>.*?)$ Preceding and trailing values. it is best to have a constant value that is searched for before and after the value being parsed.*?user (?<login>\w+\.*?)$ Look Aheads (?=[regex])[regex] (?![regex])[regex] Positive and negative look ahead allows for an initial check in the regex to see if a case is satisfied in the log messages: Example: (?=^. Example: ^. use this pattern.*?\s<sip>\s Copyright 2010 LogRhythm. MPE Rule Builder Cheat Sheet Additional Recommended Practices Rule Names • When the log message the rule will match contains a vendor message ID such as an event ID in Windows Event Logs. o Ex. EVID 528 : Failed Authentication : Bad Username or Password Common Event Names • Common Events should be generically named so that they can be re-used for a wide variety of devices. o Good: FTP Connection Succeeded o Bad: Gene6 FTP Connection Succeeded • Common Event names should always have the first letter of each word capitalized. Copyright 2010 LogRhythm. • All rule names should contain a brief description of the action described by the log. This makes searching for the rule easier and also makes the rule more descriptive of the log it matches. it is good to include the ID in the name of the rule. if a common event is being created for a log message that describes a successful connection to an FTP server. the service that generated the log message should be included in the rule name. the common event should be named so that the FTP server type is irrelevant. such as the Windows Application Event Log. Inc Page 11 of 11 . For example. This is to make the viewing of Common Events in analysis tools more consistent. • If the rule matches a log from a logging system that generates logs for a wide variety of services.