Nmap Security Scanner
*Ref Guide
Security Lists
Security Tools
Site News
Advertising
About/Contact
Credits
Sponsors





Intro Reference Guide Book Install Guide
Download Changelog Zenmap GUI Docs
Bug Reports OS Detection Propaganda Related Projects
In the Movies In the News
Lua Extensions
Prev Chapter 9. Nmap Scripting Engine Next

Lua Extensions

In addition to the significant built-in capabilities of Lua, we have written or integrated several extensions to make NSE scripts more powerful and convenient to write. These modules are compiled and installed along with Nmap. They have their own directory, nselib, which is installed in the configured datadir. Scripts need only require the default modules in order to use them. The default modules are described in the following sections.

Bitwise Logical Operations

Lua does not provide bitwise logical operations. Since they are often useful for low-level network communication, Reuben Thomas' bitwise operation library for Lua has been integrated into NSE. The arguments to the bitwise operation functions should be integers. The number of bits available for logical operations depends on the data type used to represent Lua numbers—this is typically 8-byte IEEE floats, which give 53 bits (the size of the mantissa). This implies that the bitwise operations won't work (as expected) for numbers larger than 1014. You can use them with 32-bit wide numbers without any problems. Operations involving 64-bit wide numbers, however, may not return the expected result. The logical operations start with “b” (for bit) to avoid clashing with reserved words; although xor isn't a reserved word, it seemed better to use bxor for consistency. In NSE the bitwise functions are in the bit namespace.

bit.bnot(a)

Returns the one's complement of a.

bit.band(w1,...)

Returns the bitwise and of the w's.

bit.bor(w1,...)

Returns the bitwise or of the w's.

bit.bxor(w1,...)

Returns the bitwise xor of the w's.

bit.lshift(a,b)

Returns a shifted left b places—padded with zeros.

bit.rshift(a,b)

Returns a shifted logically right b places.

bit.arshift(a,b)

Returns a shifted arithmetically right b places.

bit.mod(a,b)

Returns the integer remainder of a divided by b.

Perl Compatible Regular Expressions

One of Lua's quirks is its string patterns. While they have great performance and are tightly integrated into the Lua interpreter, they are very different in syntax and not as powerful as standard regular expressions. So we have integrated Perl compatible regular expressions into Lua using libPCRE and a modified version of the Lua PCRE library written by Reuben Thomas and Shmuel Zeigerman. These are the same sort of regular expressions used by Nmap version detection. The main modification to their library is that the NSE version only supports PCRExpressions instead of both PCRE and POSIX patterns. In order to maintain a high script execution speed, the library interfacing with libPCRE is kept very thin. It is not integrated as seamlessly as the Lua string pattern API. This allows script authors to decide when to use PCRE expressions versus Lua patterns. PCRE involves a separate pattern compilation step, which saves execution time when patterns are reused. Compiled patterns can be cached in the NSE registry and reused by other scripts. The PCRE functions reside inside the pcre namespace.

[Warning]Warning

LibPCRE has a history of security vulnerabilities allowing attackers who are able to compile arbitrary regular expressions to execute arbitrary code. More such vulnerabilities may be discovered in the future. These have never affected Nmap because it doesn't give attackers any control over the regular expressions it uses. Similarly, NSE scripts should never build regular expressions with untrusted network input. Matching hardcoded regular expressions against the untrusted input is fine.

The following documentation is derived from that supplied by the PCRE Lua lib.

pcre.new(pattern, flags, locale)

Returns a compiled regular expression. The first argument is a string describing the pattern, such as ^foo$. The second argument is a number describing which compilation flags are set. The compilation flags are set bitwise. If you want to set the 3rd (corresponding to the number 4) and the 1st (corresponding to 1) bit for example you would pass the number 5 as a second argument. The compilation flags accepted are those of the PCRE C library. These include flags for case insensitive matching (1), matching line beginnings (^) and endings ($) even in multiline strings (i.e. strings containing “\n”) (2) and a flag for matching across line boundaries (4). No compilation flags yield a default value of 0. The third (optional) argument is a string describing the locale which should be used to compile the regular expression. The variable is a string which is passed to the C standard library function setlocale. For more information on this argument refer to the documentation of setlocale. The resulting compiled regular expression is ready to be matched against strings. Compiled regular expressions are subject to Lua's garbage collection. Generally speaking, my_regex = pcre.new("pcre-pattern",0,"C") should do the job most of the time.

pcre.flags()

Returns a table of the available PCRE option flags (numbers) keyed by their names (strings). Possible names of the available strings can be retrieved from the documentation of the PCRE library used to link against Nmap. The key is the option name in the manual minus the PCRE prefix. PCRE_CASELESS becomes CASELESS for example.

pcre.version()

Returns the version of the PCRE library in use as a string. For example 6.4 05-Sep-2005.

pcre_obj:match(string, start, flags)

Returns the start point and the end point point of the first match of the compiled regular expression pcre_obj in the string. A third returned value is a table which contains false in the positions where the pattern did not match. If named sub-patterns were used the table also contains substring matches keyed by their sub-pattern name. Should no match be found the function returns nil. The second and third arguments are optional. The second argument is a number specifying where the engine should start trying to apply the pattern. The third argument specifies execution flags for the pattern. If you want to see if a given string matches a certain expression you could use:

s = pcre_obj:match("string to be searched", 0,0);
if(s) code_to_be_done_on_match end
pcre_obj:exec(string, start, flags)

This function is like match() except that a table returned as a third result contains offsets of substring matches rather than substring matches themselves. That table will not contain string keys, even if named sub-patterns are used. For example, if the whole match is at offsets 10, 20 and substring matches are at offsets 12, 14 and 16, 19 then the function returns the following: 10, 20, {12,14,16,19}

pcre_obj:gmatch(string, func, n, ef)

Tries to match the regular expression pcre_obj against string up to n times (or as many as possible if n is either not given or is not a positive number), subject to execution flags ef. Each time there is a match, func is called as func(m, t), where m is the matched string and t is a table of substring matches. This table contains false in the positions where the corresponding sub-pattern did not match. If named sub-patterns are used then the table also contains substring matches keyed by their correspondent sub-pattern names (strings). If func returns a true value, then gmatch immediately returns; gmatch returns the number of matches made.

IP Operations

The ipOps module provides some functions for manipulating IPv4 addresses. The functions reside inside the ipOps namespace.

bool = ipOps.isPrivate("ip-string")

checks whether an IP address, provided as a string in dotted-quad notation, is part of the non-routed private IP address space, as described in RFC 1918 These addresses are the well known 10.0.0.0/8,192.168.0.0/16 and 172.16.0.0/12 networks.

DWORD = ipOps.todword("ip-string")

returns the IP address as DWORD value (i.e. the IP a.b.c.d becomes (((a*256+b)*256+c)*256+d) )

a,b,c,d = ipOps.get_parts_as_number("ip-string")

returns 4 numbers corresponding to the fields in dotted-quad notation. For example, ipOps.get_parts_as_number("192.168.1.1") returns 192,168,1,1.

Short Portrules

Since portrules are mostly the same for many scripts, the shortport module provides functions for the most common tests. The arguments in brackets ([]) are optional. If no proto is provided, tcp is used. The default state is open

shortport.portnumber(port,[proto],[state])

The port argument is either a number or a table of numbers which are interpreted as port numbers, against which the script should run.

shortport.service(service,[proto],[state])

The service argument is either a string or a table of strings which are interpreted as service names (e.g. "http", "https", "smtp" or "ftp") against which the script should run. These service names are determined by Nmap's version scan or (if no version scan information is available) the service assigned to the port in nmap-services (i.e. "http" for TCP port 80).

shortport.port_or_service(port,service,[proto],[state])

This is a combination of the above functions, since many scripts explicitly try to run against the well known ports, but want also to run against any other port which was discovered to run the named service. A typical example for this function is: portrule = shortport.port_or_service(22,"ssh").

Functional Programming Style List Operations

People used to programming in functional languages, such as Lisp or Haskell appreciate their handling of lists very much. The listop module tries to bring much of the functionality from functional languages to Lua using Lua's central data structure, the table, as a base for its list operations. Highlights include a map function applying a given function to each element of a list.

bool = listop.is_empty(list)

Returns true if the given list is empty.

bool = listop.is_list(value)

Returns true if the given value is a list (or rather a table).

list = listop.map(function, list)

The provided function is applied to each element of the list separately. The returned list contains the results of each function call. For example listop.map(tostring,{1,2,true}) returns {"1","2","true"}.

value = listop.apply(function, list)

All of the elements in the list are passed to a call of function. The result is then returned. For example listop.apply(math.max,{1,5,6,7,50000}) yields 50000.

list = listop.filter(predicate, list)

Returns a list containing only those elements for which the predicate returns true. The predicate has to be a function, which takes an element of the list as argument and the result of which is interpreted as boolean value. If it returns true (or rather anything besides false and nil) the argument is appended to the return value of filter. For example: listop.filter(isnumber,{1,2,3,"foo",4,"bar"}) returns {1,2,3,4}.

list = listop.flatten(list)

Since a list can itself contain lists as elements, flatten returns a list which only contains values that are not themselves lists. For example: listop.flatten({1,2,3,"foo",{4,5,{"bar"}}}) returns {1,2,3,"foo",4,5,"bar"}.

list = listop.append(list1, list2)

Returns a list containing all elements of list1 appended by all elements of list2.

list = listop.cons(value1, value2)

Returns a list containing value1 appended by value2, which may be of any type.

list = listop.reverse(list)

Returns a list containing all elements of the given list in inverted order.

value = listop.car(list)

Returns the first element of the given list.

value = listop.ncar(list,n)

Returns the nth (or first if n is omitted) element of the given list.

value = listop.cdr(list)

Returns a list containing all elements but the first of the given list.

value = listop.ncdr(list, n)

Returns a list containing all elements but the first n of the given list, where n is 2 if it is omitted.

String Buffer Operations

Lua's string operations are very flexible and offer an easy-to-use way to manipulate strings. Concatenation using the .. operator is such an operation. The drawback of the built-in API however is the way it handles concatenation of many string values. Since strings in Lua are immutable values, each time you concatenate two strings both get copied into the result string. The strbuf module offers a workaround for this problem, while maintaining the nice syntax. This is accomplished by overloading the concatenation operator (..) the equality operator (==) and the tostring operator. By overloading these operators, we reduce the overhead of using a string buffer instead of a plain string to wrap the first literal string assigned to a variable inside a strbuf.new() call. Afterwards you can append to the string buffer, or compare two string buffers for equality just as you would do with normal strings. When looking at the details there are some more restrictions/oddities: The concatenation operator requires its left-hand value to be a string buffer. Therefore, if you want to prepend a string to a given string buffer you have to create a new string buffer out of the string you want to prepend. The string buffer's tostring operator concatenates the strings inside the buffer using newlines by default, since this appears to be the separator used most often.

buffer = strbuf.new("first-string")

Creates a new string buffer. The argument is optional and is the first string to be added to the buffer.

buffer = strbuf.concat(strbuf1, value)

Concatenates the value (which has to be either a string or a string buffer) to strbuf1. This is also the function serving as the string buffer's concatenation operator. The above function call can thus also be expressed as: buffer = strbuf1 .. value

bool = strbuf.eqbuf(strbuf1, strbuf2)

Compares strbuf1 and strbuf2 for equality. For the function to return true, both values must be string buffers containing exactly the same strings. The eqbuf function is called to compare two strings for equality.

strbuf.clear(strbuf)

Deletes all strings in strbuf.

string = strbuf.dump(strbuf, "delimiter")

Dumps strbuf's contents as string. The second parameter is used as a delimiter between the strings stored inside strbuf. dump(strbuf, "\n") is used as the tostring function of string buffers.

URL Manipulation Functions

URL manipulation functions have obvious uses. Fortunately there is already an implementation of URL generation functions inside the Lua-socket package, which is fairly complete and well documented For NSE, Lua-socket's URL module was extended with two functions:

table = url.parse_query("query-string")

This function takes a query-string of the form name1=value1&name2=value2... and returns a table containing the name-value pairs, with the name as the key and the value as its associated value. The table corresponding to the above query-string would have two entries: table["name1"]="value1" and table["name2"]="value2".

query_string = url.build_query(table)

This is the inverse function to parse_query().

Buffered Network I/O Helper Functions

The match module was written to provide functions which can be used for delimiting data received by the receive_buf() function from the Network I/O API:

start,end = match.regex("regexpattern")

This is actually a wrapper around NSE' PCRE library exec function (see the section called “Perl Compatible Regular Expressions”, thus giving script developers the possibility to use regular expressions for delimiting instead of Lua's string patterns. If you want to get the data in chunks separated by regex (which has to be a valid regular expression), you would write status, val = sockobj:receive_buf(match.lua("regex")).

start,end = match.numbytes(number)

Takes a number as argument and returns that many bytes. It can be used to get a buffered version of sockobj:receive_bytes(n) in case a script requires more than one fixed-size chunk, as the unbuffered version may return more bytes than requested and thus would require you to do the parsing on your own.

HTTP Functions

The http module provides functions for dealing with the client side of the http protocol. The functions reside inside the http namespace. The return value of each function in this module is a table with the following keys: status, header and body. status is the status code of the http request In case of an error status is nil. header is a table with the headers received from the server. The header names are lower-cased and multiple headers of the same name are concatenated with comma. body holds a string with the request body.

table = http.get(host,port,path,[options])

Fetches a resource with a GET request. The first argument is either a string with the hostname or a table like the host table passed by nmap. The second argument is either the port number or a table like the port table passed by nmap. The third argument is the path of the resource. The fourth argument is a table for further options. The table may have 2 keys: timeout and header. timeout is the timeout used for the socket operations. header is a table with additional headers to be used for the request. The function builds the request and calls http.request

table = http.request(host,port,request,[options])

Sends request to host:port and parses the answer. The first argument is either a string with the hostname or a table like the host table passed by nmap. The second argument is either the port number or a table like the port table passed by nmap. SSL is used for the request if either port.service equals "https" or port.version.service_tunnel equals "ssl". The third argument is the request. The fourth argument is a table for further options. You can specify a timeout for the socket operations with the timeout key.

table = http.get_url(url,[options])

Parses url and calls http.get with the result. The second argument is a table for further options. The table may have 2 keys: timeout and header. timeout is the timeout used for the socket operations. header is a table with additional headers to be used for the request.

Data File Parsing Functions

The datafiles module provides functions for reading and parsing Nmap's data files (e.g. nmap-protocol, nmap-rpc, etc.). These functions' return values are setup for use with exception handling via nmap.new_try().

bool, table = datafiles.parse_protocols()

This function reads and parses Nmap's nmap-protocols file. bool is a boolen value indicating success. If bool is true, then the second returned value is a table with protocol numbers indexing the protocol names. If bool is false, an error message is returned as the second value instead of the table.

bool, table = datafiles.parse_rpc()

This function reads and parses Nmap's nmap-rpc file. bool is a boolen value indicating success. If bool is true, then the second returned value is a table with RPC numbers indexing the RPC names. If bool is false, an error message is returned as the second value instead of the table.

bool, table = datafiles.parse_services([protocol])

This function reads and parses Nmap's nmap-services file. bool is a boolen value indicating success. If bool is true, then the second returned value is a table containing two other tables: tcp{} and udp{}. tcp{} contains services indexed by TCP port numbers. udp{} is the same, but for UDP. You can pass "tcp" or "udp" as an argument to parse_services() to only get the corresponding table. If bool is false, an error message is returned as the second value instead of the table.

Various Utility Functions

The stdnse library contains various handy functions which are too small to justify modules of their own:

stdnse.print_debug(...)

Wrapper function around print_debug_unformatted() in the nmap namespace. The first argument, if numeric, is used as the necessary debug level to print the message (it defaults to 1 if omitted). All remaining arguments are processed with Lua's string.format() function, which provides a C-style printf interface.

table = stdnse.strsplit("delimiter","text")

This function will certainly be appreciated by Perl programmers. It takes two strings as arguments and splits the second one around all occurrences of the first one, returning a table, which contains the substrings without the delimiting string.

string = stdnse.strjoin("delimiter", table)

Inverse function to strsplit(). Basically this is Lua's table.concat() function with the parameters swapped for coherence.


Prev Up Next
Script Language Home Nmap API
[ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]