Nmap Security Scanner
*Ref Guide
Security Lists
Security Tools
Site News
Advertising
About/Contact
Credits
Sponsors





Intro Reference Guide Book Install Guide
Download Changelog Zenmap GUI Docs
Bug Reports OS Detection Propaganda Related Projects
In the Movies In the News
Implementation
Prev Chapter 9. Nmap Scripting Engine Next

Implementation

We don't need a dozen pages of low-level trivial details, but it would be nice to have a few sections describing notable aspects of the NSE implementation (maybe things like how the parallelization algorithms work, how Lua is embedded, performance related notes. Information which might help script writers is particularly desirable. I tend to think reasons for choosing Lua may be better suited to the section called “Lua Base Language”, but it could be placed here instead.

Now how does all this work? The following section describes some interesting aspects of the NSE. While the focus primarily lies on giving script writers a better feeling of what happens with scripts, it should also provide a starting point for understanding (and extending) the NSE sources.

Initialization Phase

During its initialization stage, Nmap loads the Lua interpreter and its provided libraries get loaded. These libraries are:

  • The package library (namespace: package)—Lua's package-lib provides (among others) the require function, used to load modules from the nselib.

  • The table library (namespace: table)—The table manipulation library contains many functions used to operate on tables—Lua's central data structure.

  • The I/O library (namespace: io)—The Input/Output library offers functions such as reading files and reading the output from programs you execute.

  • The OS library (namespace: os)—The Operating System lib provides facilities of the operating system, including filesystem operations (renaming/removing files, creating of temporary filenames) and access to the environment.

  • The string library (namespace: string)—The string library helps you with functions used to manipulate strings inside Lua. Functions include: printf-style string formating, pattern matching using Lua-style patterns, substring extraction, etc.

  • The math library (namespace: math)—Since usually numbers in Lua correspond to the double C-type, the math library gives you access to rounding functions, trigonometric functions, random number generation, and many more.

  • The debug library (namespace: debug)—The debug library provides you with a somewhat lower level API to the Lua-interpreter. Through it you can access functions along the execution stack, get function closures and object metatables, etc.

In addition to loading the libraries provided with Lua, the functions in the nmap namespace also get loaded. and search path for modules is set to the default one prepended by the nselib directory (which is searched in the locations Nmap searches for its data files and scripts). In this step the provided script arguments also get stored inside the registry.

The next phase of NSE initialization is loading the chosen scripts, which are the arguments provided to the --script option or safe,intrusive, in case of a default script scan. The string version is appended, if version detection was enabled. The arguments afterwards are tried to be interpreted as script categories. This is done via a short Lua function hard-coded into nse_init.cc called Entry. If you take a look into the script.db you'll see that the Entry lines inside it are Lua function calls with a table as argument. The arguments that didn't produce any filenames are then interpreted as file or directory names themselves. If this also fails, the script scan is aborted.

In the next stage the found files are loaded as chunks, each with its own environment, having read but not write access to the global name space and saved inside two globally accessible Lua tables: hosttests and porttests depending on the type of script. Because scripts only get loaded once, values stored inside variables during a script's execution against one host or port can be accessed when the same script runs against another target. This can be used to save computation time when a script is run against multiple targets. See Example 9.3, “Using local variables to save data.” During this stage scripts are also are also provided with a default runlevel (1.0), if they don't specify one themselves and a check is performed whether they contain an action and a description field.

Example 9.3. Using local variables to save data.

id="persistent locals example"
description="This sample script shows how data can be stored across \
several invocations of a script against multiple targets"
author="Stoiko Ivanov"
categories = {"safe"}

require "shortport"
portrule = shortport.portnumber(80)
-- we have to declare the variable in the script's global scope
-- because if we declare it inside the action it would get redefined
-- with each call to the action 
local filecontent = nil
require "strbuf"
action= function(host, port)
  if(filecontent == nil) then
    filecontent = strbuf.new()
    for line in io.lines("a_filename_we_want_to_read_from")
      filecontent = filecontent .. line
    end
  end
  --rest of the script doing something with the filecontent, we just
  --read
end

Matching of Scripts to Targets

After the initialization is finished the hostrules and portrules are evaluated for each host in the current target group. At this check a list is built which contains the combinations of scripts and the hosts they will run against. It should be noted that the rules of all chosen scripts are checked against all hosts and their open and open|filtered ports. Therefore it is advisable to leave the rules as simple as possible and to do all the computation inside the action, as a script will only be executed if it is run against a specific target. After the check those script-target combinations get their own Lua-thread which is anchored in Lua's C-API registry to prevent their garbage collection. These thread_records are afterwards sorted by run level and all script-target combinations of one run level are stored in a list, in order to ensure that scripts with a higher run level are run after those with a lower one.

Running Scripts

Now to the actual script scanning, and the way NSE accomplishes parallelization. Lua, through its concept of coroutines offers collaborative multi-threading, which means that scripts can suspend themselves, at defined points, and let other coroutines execute. Since network I/O, especially waiting for responses from remote host, is the part of scripts which would consume most time with waiting, this is the point where scripts suspend themselves and let others execute. Each call to some of the functions of the Nsock wrapper causes the calling script to yield (pause). Once the request is processed by the Nsock library, the callback causes the script to be pushed from the waiting queue to the running queue, which will eventually let it resume its operation.

Adding C Modules to Nselib

This section tries to give a short walk-through to adding nselib modules written in C (or C++) to Nmap's build system, since this has shown to be sometimes tedious. Writing C modules is described at length in Programming in Lua, Second Edition Basically C modules consist of the functions they provide to Lua, which have to be of type lua_CFunction Additionally they have to contain a function which is used to actually open the module. By convention these function names are luaopen_modulename. A good starting point for writing such modules is provided with bit.c and pcre.c inside the nselib/ subdirectory of Nmap's source tree, which are two C modules already provided by the nselib. C modules basically are shared libraries which get loaded at runtime by Lua.

The Unix build system uses libtool for compilation in a platform independent way. As long as the new module does not depend on foreign libraries, you should only need to add modulename.so to the all and clean targets in Makefile.in and copy and adapt the lines from bit.so. If your module does have dependencies you will most probably have to add checks for those libraries to configure.ac and put the dependencies inside the libtool commands in Makefile.in—here you should take a look at how pcre.so handles this. So much for the way it should work. Now for some pitfalls we've come across so far: These are results from building shared libraries in general and not really specific to nselib. Linking with static libraries (e.g. libnbase) sometimes leads to problems with exporting symbols on some platforms (in our case this was a x86_64-linux platform). To our knowledge no such problems occur when linking against already existing shared libraries.

The Windows build system requires C module developers to create a MS Visual Studio Project file for their module (<modulename>.vcproj) inside the nselib subdirectory. On Windows you have to include the liblua/ subdirectory as an additional include path as well as a library search path. In addition you have to tell the project to link against the liblua.lib static library provided with Nmap. Other properties of the project should be the same as for other nselib C modules (e.g. see nse_bitlib.vcproj). Afterwards you have to include the newly created project file in Nmap's Visual Studio solution file (mswin32\nmap.sln) and make sure that nse_bitlib.vcproj depends on your project, because it is there nselib-modules get copied to their final destinations and are included in Nmap.


Prev Up Next
Example Scripts Home NSE Script License and Community Contributions
[ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]