Nmap Security Scanner Ref Guide Security Lists Security Tools Site News Advertising About/Contact Credits Sponsors |
Intro | Reference Guide | Book | Install Guide
Download | Changelog | Zenmap GUI | Docs
Bug Reports | OS Detection | Propaganda | Related Projects
In the Movies | In the News
Lua Extensions
| Chapter 9. Nmap Scripting Engine |
## Lua Extensions In addition to the significant built-in capabilities of Lua, we have written or integrated several extensions to make NSE scripts more powerful and convenient to write. These modules are compiled and installed along with Nmap. They have their own directory, `nselib`, which is installed in the configured datadir. Scripts need only `require` the default modules in order to use them. The default modules are described in the following sections. ### Bitwise Logical Operations Lua does not provide bitwise logical operations. Since they are often useful for low-level network communication, Reuben Thomas' bitwise operation library for Lua has been integrated into NSE. The arguments to the bitwise operation functions should be integers. The number of bits available for logical operations depends on the data type used to represent Lua numbers—this is typically 8-byte IEEE floats, which give 53 bits (the size of the mantissa). This implies that the bitwise operations won't work (as expected) for numbers larger than 1014. You can use them with 32-bit wide numbers without any problems. Operations involving 64-bit wide numbers, however, may not return the expected result. The logical operations start with “b” (for `bit`) to avoid clashing with reserved words; although `xor` isn't a reserved word, it seemed better to use `bxor` for consistency. In NSE the bitwise functions are in the `bit` namespace. `bit.bnot(a)` Returns the one's complement of a. `bit.band(w1,...)` Returns the bitwise `and` of the w's. `bit.bor(w1,...)` Returns the bitwise `or` of the w's. `bit.bxor(w1,...)` Returns the bitwise `xor` of the w's. `bit.lshift(a,b)` Returns a shifted left b places—padded with zeros. `bit.rshift(a,b)` Returns a shifted logically right b places. `bit.arshift(a,b)` Returns a shifted arithmetically right b places. `bit.mod(a,b)` Returns the integer remainder of a divided by b. ### Perl Compatible Regular Expressions One of Lua's quirks is its string patterns. While they have great performance and are tightly integrated into the Lua interpreter, they are very different in syntax and not as powerful as standard regular expressions. So we have integrated Perl compatible regular expressions into Lua using libPCRE and a modified version of the Lua PCRE library written by Reuben Thomas and Shmuel Zeigerman. These are the same sort of regular expressions used by Nmap version detection. The main modification to their library is that the NSE version only supports PCRExpressions instead of both PCRE and POSIX patterns. In order to maintain a high script execution speed, the library interfacing with libPCRE is kept very thin. It is not integrated as seamlessly as the Lua string pattern API. This allows script authors to decide when to use PCRE expressions versus Lua patterns. PCRE involves a separate pattern compilation step, which saves execution time when patterns are reused. Compiled patterns can be cached in the NSE registry and reused by other scripts. The PCRE functions reside inside the `pcre` namespace.
| Warning
LibPCRE has a history of security vulnerabilities allowing attackers who are able to compile arbitrary regular expressions to execute arbitrary code. More such vulnerabilities may be discovered in the future. These have never affected Nmap because it doesn't give attackers any control over the regular expressions it uses. Similarly, NSE scripts should never build regular expressions with untrusted network input. Matching hardcoded regular expressions against the untrusted input is fine.
  The following documentation is derived from that supplied by the PCRE Lua lib. `pcre.new(pattern, flags, locale)` Returns a compiled regular expression. The first argument is a string describing the pattern, such as `^foo$`. The second argument is a number describing which compilation flags are set. The compilation flags are set bitwise. If you want to set the 3rd (corresponding to the number 4) and the 1st (corresponding to 1) bit for example you would pass the number 5 as a second argument. The compilation flags accepted are those of the PCRE C library. These include flags for case insensitive matching (1), matching line beginnings (^) and endings ($) even in multiline strings (i.e. strings containing “\n”) (2) and a flag for matching across line boundaries (4). No compilation flags yield a default value of 0. The third (optional) argument is a string describing the locale which should be used to compile the regular expression. The variable is a string which is passed to the C standard library function `setlocale`. For more information on this argument refer to the documentation of `setlocale`. The resulting compiled regular expression is ready to be matched against strings. Compiled regular expressions are subject to Lua's garbage collection. Generally speaking, `my_regex = pcre.new("`pcre-pattern`",0,"C")` should do the job most of the time. `pcre.flags()` Returns a table of the available PCRE option flags (numbers) keyed by their names (strings). Possible names of the available strings can be retrieved from the documentation of the PCRE library used to link against Nmap. The key is the option name in the manual minus the `PCRE` prefix. `PCRE_CASELESS` becomes `CASELESS` for example. `pcre.version()` Returns the version of the PCRE library in use as a string. For example `6.4 05-Sep-2005`. `pcre_obj:match(string, start, flags)` Returns the start point and the end point point of the first match of the compiled regular expression pcre_obj in the string. A third returned value is a table which contains `false` in the positions where the pattern did not match. If named sub-patterns were used the table also contains substring matches keyed by their sub-pattern name. Should no match be found the function returns `nil`. The second and third arguments are optional. The second argument is a number specifying where the engine should start trying to apply the pattern. The third argument specifies execution flags for the pattern. If you want to see if a given string matches a certain expression you could use:
      s = pcre_obj:match("string to be searched", 0,0);
      if(s) code_to_be_done_on_match end
`pcre_obj:exec(string, start, flags)` This function is like `match()` except that a table returned as a third result contains offsets of substring matches rather than substring matches themselves. That table will not contain string keys, even if named sub-patterns are used. For example, if the whole match is at offsets `10, 20` and substring matches are at offsets `12, 14` and `16, 19` then the function returns the following: `10, 20, {12,14,16,19}` `pcre_obj:gmatch(string, func, n, ef)` Tries to match the regular expression `pcre_obj` against `string` up to `n` times (or as many as possible if `n` is either not given or is not a positive number), subject to execution flags ef. Each time there is a match, `func` is called as `func(m, t)`, where `m` is the matched string and `t` is a table of substring matches. This table contains `false` in the positions where the corresponding sub-pattern did not match. If named sub-patterns are used then the table also contains substring matches keyed by their correspondent sub-pattern names (strings). If `func` returns a `true` value, then gmatch immediately returns; gmatch returns the number of matches made. ### IP Operations The `ipOps` module provides some functions for manipulating IPv4 addresses. The functions reside inside the `ipOps` namespace. `bool = ipOps.isPrivate("ip-string")` checks whether an IP address, provided as a string in dotted-quad notation, is part of the non-routed private IP address space, as described in RFC 1918 These addresses are the well known `10.0.0.0/8`,`192.168.0.0/16` and `172.16.0.0/12` networks. `DWORD = ipOps.todword("ip-string")` returns the IP address as DWORD value (i.e. the IP `a.b.c.d` becomes `(((a*256+b)*256+c)*256+d)` ) `a,b,c,d = ipOps.get_parts_as_number("ip-string")` returns 4 numbers corresponding to the fields in dotted-quad notation. For example, `ipOps.get_parts_as_number("192.168.1.1")` returns `192,168,1,1`. ### Short Portrules Since portrules are mostly the same for many scripts, the `shortport` module provides functions for the most common tests. The arguments in brackets (`[]`) are optional. If no `proto` is provided, `tcp` is used. The default `state` is `open` `shortport.portnumber(port,[proto],[state])` The port argument is either a number or a table of numbers which are interpreted as port numbers, against which the script should run. `shortport.service(service,[proto],[state])` The service argument is either a string or a table of strings which are interpreted as service names (e.g. `"http"`, `"https"`, `"smtp"` or `"ftp"`) against which the script should run. These service names are determined by Nmap's version scan or (if no version scan information is available) the service assigned to the port in `nmap-services` (i.e. "http" for TCP port 80). `shortport.port_or_service(port,service,[proto],[state])` This is a combination of the above functions, since many scripts explicitly try to run against the well known ports, but want also to run against any other port which was discovered to run the named service. A typical example for this function is: `portrule = shortport.port_or_service(22,"ssh")`. ### Functional Programming Style List Operations People used to programming in functional languages, such as Lisp or Haskell appreciate their handling of lists very much. The `listop` module tries to bring much of the functionality from functional languages to Lua using Lua's central data structure, the table, as a base for its list operations. Highlights include a `map` function applying a given function to each element of a list. `bool = listop.is_empty(list)` Returns `true` if the given list is empty. `bool = listop.is_list(value)` Returns `true` if the given value is a list (or rather a table). `list = listop.map(function, list)` The provided function is applied to each element of the list separately. The returned list contains the results of each function call. For example `listop.map(tostring,{1,2,true})` returns `{"1","2","true"}`. `value = listop.apply(function, list)` All of the elements in the list are passed to a call of `function`. The result is then returned. For example `listop.apply(math.max,{1,5,6,7,50000})` yields `50000`. `list = listop.filter(predicate, list)` Returns a list containing only those elements for which the predicate returns true. The predicate has to be a function, which takes an element of the list as argument and the result of which is interpreted as boolean value. If it returns true (or rather anything besides `false` and `nil`) the argument is appended to the return value of `filter`. For example: `listop.filter(isnumber,{1,2,3,"foo",4,"bar"})` returns `{1,2,3,4}`. `list = listop.flatten(list)` Since a list can itself contain lists as elements, `flatten` returns a list which only contains values that are not themselves lists. For example: `listop.flatten({1,2,3,"foo",{4,5,{"bar"}}})` returns `{1,2,3,"foo",4,5,"bar"}`. `list = listop.append(list1, list2)` Returns a list containing all elements of list1 appended by all elements of `list2`. `list = listop.cons(value1, value2)` Returns a list containing `value1` appended by `value2`, which may be of any type. `list = listop.reverse(list)` Returns a list containing all elements of the given list in inverted order. `value = listop.car(list)` Returns the first element of the given list. `value = listop.ncar(list,n)` Returns the nth (or first if n is omitted) element of the given list. `value = listop.cdr(list)` Returns a list containing all elements but the first of the given list. `value = listop.ncdr(list, n)` Returns a list containing all elements but the first n of the given list, where n is 2 if it is omitted. ### String Buffer Operations Lua's string operations are very flexible and offer an easy-to-use way to manipulate strings. Concatenation using the `..` operator is such an operation. The drawback of the built-in API however is the way it handles concatenation of many string values. Since strings in Lua are immutable values, each time you concatenate two strings both get copied into the result string. The `strbuf` module offers a workaround for this problem, while maintaining the nice syntax. This is accomplished by overloading the concatenation operator (`..`) the equality operator (`==`) and the tostring operator. By overloading these operators, we reduce the overhead of using a string buffer instead of a plain string to wrap the first literal string assigned to a variable inside a `strbuf.new()` call. Afterwards you can append to the string buffer, or compare two string buffers for equality just as you would do with normal strings. When looking at the details there are some more restrictions/oddities: The concatenation operator requires its left-hand value to be a string buffer. Therefore, if you want to prepend a string to a given string buffer you have to create a new string buffer out of the string you want to prepend. The string buffer's `tostring` operator concatenates the strings inside the buffer using newlines by default, since this appears to be the separator used most often. `buffer = strbuf.new("first-string")` Creates a new string buffer. The argument is optional and is the first string to be added to the buffer. `buffer = strbuf.concat(strbuf1, value)` Concatenates the `value` (which has to be either a string or a string buffer) to `strbuf1`. This is also the function serving as the string buffer's concatenation operator. The above function call can thus also be expressed as: `buffer = strbuf1 .. value` `bool = strbuf.eqbuf(strbuf1, strbuf2)` Compares `strbuf1` and `strbuf2` for equality. For the function to return `true`, both values must be string buffers containing exactly the same strings. The `eqbuf` function is called to compare two strings for equality. `strbuf.clear(strbuf)` Deletes all strings in `strbuf`. `string = strbuf.dump(strbuf, "delimiter")` Dumps `strbuf`'s contents as string. The second parameter is used as a delimiter between the strings stored inside `strbuf`. `dump(strbuf, "\n")` is used as the `tostring` function of string buffers. ### URL Manipulation Functions URL manipulation functions have obvious uses. Fortunately there is already an implementation of URL generation functions inside the Lua-socket package, which is fairly complete and well documented For NSE, Lua-socket's URL module was extended with two functions: `table = url.parse_query("query-string")` This function takes a `query-string` of the form `name1=value1&name2=value2...` and returns a table containing the name-value pairs, with the `name` as the key and the `value` as its associated value. The table corresponding to the above `query-string` would have two entries: `table["name1"]="value1"` and `table["name2"]="value2"`. `query_string = url.build_query(table)` This is the inverse function to `parse_query()`. ### Buffered Network I/O Helper Functions The `match` module was written to provide functions which can be used for delimiting data received by the `receive_buf()` function from the Network I/O API: `start,end = match.regex("regexpattern")` This is actually a wrapper around NSE' PCRE library `exec` function (see the section called “Perl Compatible Regular Expressions”, thus giving script developers the possibility to use regular expressions for delimiting instead of Lua's string patterns. If you want to get the data in chunks separated by `regex` (which has to be a valid regular expression), you would write `status, val = sockobj:receive_buf(match.lua("regex"))`. `start,end = match.numbytes(number)` Takes a number as argument and returns that many bytes. It can be used to get a buffered version of `sockobj:receive_bytes(n)` in case a script requires more than one fixed-size chunk, as the unbuffered version may return more bytes than requested and thus would require you to do the parsing on your own. ### HTTP Functions The `http` module provides functions for dealing with the client side of the http protocol. The functions reside inside the `http` namespace. The return value of each function in this module is a table with the following keys: `status`, `header` and `body`. `status` is the status code of the http request In case of an error status is `nil`. `header` is a table with the headers received from the server. The header names are lower-cased and multiple headers of the same name are concatenated with comma. `body` holds a string with the request body. `table = http.get(host,port,path,[options])` Fetches a resource with a `GET` request. The first argument is either a string with the hostname or a table like the host table passed by nmap. The second argument is either the port number or a table like the port table passed by nmap. The third argument is the path of the resource. The fourth argument is a table for further options. The table may have 2 keys: `timeout` and `header`. `timeout` is the timeout used for the socket operations. `header` is a table with additional headers to be used for the request. The function builds the request and calls `http.request` `table = http.request(host,port,request,[options])` Sends `request` to `host`:`port` and parses the answer. The first argument is either a string with the hostname or a table like the host table passed by nmap. The second argument is either the port number or a table like the port table passed by nmap. SSL is used for the request if either `port.service` equals `"https"` or `port.version.service_tunnel` equals `"ssl"`. The third argument is the request. The fourth argument is a table for further options. You can specify a timeout for the socket operations with the timeout key. `table = http.get_url(url,[options])` Parses `url` and calls `http.get` with the result. The second argument is a table for further options. The table may have 2 keys: `timeout` and `header`. `timeout` is the timeout used for the socket operations. `header` is a table with additional headers to be used for the request. ### Data File Parsing Functions The `datafiles` module provides functions for reading and parsing Nmap's data files (e.g. `nmap-protocol`, `nmap-rpc`, etc.). These functions' return values are setup for use with exception handling via `nmap.new_try()`. `bool, table = datafiles.parse_protocols()` This function reads and parses Nmap's `nmap-protocols` file. `bool` is a boolen value indicating success. If `bool` is true, then the second returned value is a table with protocol numbers indexing the protocol names. If `bool` is false, an error message is returned as the second value instead of the table. `bool, table = datafiles.parse_rpc()` This function reads and parses Nmap's `nmap-rpc` file. `bool` is a boolen value indicating success. If `bool` is true, then the second returned value is a table with RPC numbers indexing the RPC names. If `bool` is false, an error message is returned as the second value instead of the table. `bool, table = datafiles.parse_services([protocol])` This function reads and parses Nmap's `nmap-services` file. `bool` is a boolen value indicating success. If `bool` is true, then the second returned value is a table containing two other tables: `tcp{}` and `udp{}`. `tcp{}` contains services indexed by TCP port numbers. `udp{}` is the same, but for UDP. You can pass "tcp" or "udp" as an argument to `parse_services()` to only get the corresponding table. If `bool` is false, an error message is returned as the second value instead of the table. ### Various Utility Functions The `stdnse` library contains various handy functions which are too small to justify modules of their own: `stdnse.print_debug(...)` Wrapper function around `print_debug_unformatted()` in the `nmap` namespace. The first argument, if numeric, is used as the necessary debug level to print the message (it defaults to 1 if omitted). All remaining arguments are processed with Lua's `string.format()` function, which provides a C-style printf interface. `table = stdnse.strsplit("delimiter","text")` This function will certainly be appreciated by Perl programmers. It takes two strings as arguments and splits the second one around all occurrences of the first one, returning a table, which contains the substrings without the delimiting string. `string = stdnse.strjoin("delimiter", table)` Inverse function to `strsplit()`. Basically this is Lua's `table.concat()` function with the parameters swapped for coherence.
Script Language |  | Nmap API
| [ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]